[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:22:27 dist-test-slave-8w9v krb5kdc[5720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:22:27 dist-test-slave-8w9v krb5kdc[5720](info): set up 2 sockets
Aug 12 14:22:27 dist-test-slave-8w9v krb5kdc[5720](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 14:22:28.531106 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:22:28 dist-test-slave-8w9v krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:22:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:22:28.672607 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:37435
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:37435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:28.784667 5736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:28.784937 5736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:28.784965 5736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:28.786231 5736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:28.786299 5736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:28.786317 5736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:28.786332 5736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:28.786346 5736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:22:28.788486 5736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:37435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:37435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:28.789074 5736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:28.789445 5736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:28.798085 5741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:28.798117 5742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:28.798163 5736 server_base.cc:1034] running on GCE node
W20260812 14:22:28.798142 5744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:28.798815 5736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:28.799284 5736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:28.800470 5736 hybrid_clock.cc:648] HybridClock initialized: now 1786544548800455 us; error 65 us; skew 500 ppm
Aug 12 14:22:28 dist-test-slave-8w9v krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544548, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:28.803488 5736 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:22:28.804286 5736 webserver.cc:460] Webserver started at http://127.5.148.62:41825/ using document root <none> and password file <none>
I20260812 14:22:28.804634 5736 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:28.804718 5736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:28.804937 5736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:28.806384 5736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "e56fa4cd0b744a629e90e7785426ccf8"
format_stamp: "Formatted at 2026-08-12 14:22:28 on dist-test-slave-8w9v"
server_key: "ce4070c5ce31da2d8c281e87063bc960"
server_key_iv: "0b410999591a1c38f9f66cc986838937"
server_key_version: "encryptionkey@0"
I20260812 14:22:28.807133 5736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "e56fa4cd0b744a629e90e7785426ccf8"
format_stamp: "Formatted at 2026-08-12 14:22:28 on dist-test-slave-8w9v"
server_key: "ce4070c5ce31da2d8c281e87063bc960"
server_key_iv: "0b410999591a1c38f9f66cc986838937"
server_key_version: "encryptionkey@0"
I20260812 14:22:28.809268 5736 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260812 14:22:28.810483 5751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:28.810732 5736 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:28.810887 5736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "e56fa4cd0b744a629e90e7785426ccf8"
format_stamp: "Formatted at 2026-08-12 14:22:28 on dist-test-slave-8w9v"
server_key: "ce4070c5ce31da2d8c281e87063bc960"
server_key_iv: "0b410999591a1c38f9f66cc986838937"
server_key_version: "encryptionkey@0"
I20260812 14:22:28.811014 5736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:28.835918 5736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:28.838630 5736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:28.838881 5736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:28.844707 5736 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:37435
I20260812 14:22:28.844802 5810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:37435 every 8 connection(s)
I20260812 14:22:28.845469 5736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:28.847029 5811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:28.849984 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 5736
I20260812 14:22:28.850123 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:28.851089 5712 external_mini_cluster.cc:1428] Setting key e46a5aefe41bf007a60234ad2c11e34a
I20260812 14:22:28.851205 5811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8: Bootstrap starting.
I20260812 14:22:28.852670 5811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:28.853230 5811 log.cc:824] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:28.854275 5811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8: No bootstrap required, opened a new log
I20260812 14:22:28.856932 5811 raft_consensus.cc:348] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56fa4cd0b744a629e90e7785426ccf8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37435 } }
I20260812 14:22:28.857197 5811 raft_consensus.cc:374] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:28.857283 5811 raft_consensus.cc:729] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56fa4cd0b744a629e90e7785426ccf8, State: Initialized, Role: FOLLOWER
I20260812 14:22:28.857591 5811 consensus_queue.cc:260] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56fa4cd0b744a629e90e7785426ccf8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37435 } }
I20260812 14:22:28.857772 5811 raft_consensus.cc:388] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:28.857825 5811 raft_consensus.cc:482] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:28.857885 5811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:28.859520 5811 raft_consensus.cc:504] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56fa4cd0b744a629e90e7785426ccf8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37435 } }
I20260812 14:22:28.859776 5811 leader_election.cc:302] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e56fa4cd0b744a629e90e7785426ccf8; no voters:
I20260812 14:22:28.860054 5811 leader_election.cc:288] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:28.860164 5816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:28.860409 5816 raft_consensus.cc:686] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [term 1 LEADER]: Becoming Leader. State: Replica: e56fa4cd0b744a629e90e7785426ccf8, State: Running, Role: LEADER
I20260812 14:22:28.860611 5811 sys_catalog.cc:564] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:28.860617 5816 consensus_queue.cc:237] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56fa4cd0b744a629e90e7785426ccf8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37435 } }
I20260812 14:22:28.861287 5817 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e56fa4cd0b744a629e90e7785426ccf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56fa4cd0b744a629e90e7785426ccf8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37435 } } }
I20260812 14:22:28.861299 5818 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e56fa4cd0b744a629e90e7785426ccf8. Latest consensus state: current_term: 1 leader_uuid: "e56fa4cd0b744a629e90e7785426ccf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56fa4cd0b744a629e90e7785426ccf8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37435 } } }
I20260812 14:22:28.861479 5817 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:28.861531 5818 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8 [sys.catalog]: This master's current role is: LEADER
W20260812 14:22:28.863732 5835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:22:28.863818 5835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:22:28.863925 5824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:22:28.864431 5824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 14:22:28 dist-test-slave-8w9v krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:28.867525 5824 catalog_manager.cc:1347] Generated new cluster ID: 96697a7b39cf4a2fb9a213329b673571
I20260812 14:22:28.867594 5824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:22:28.873019 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:28.852802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.5.148.62:37435)
0812 14:22:28.853183 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:28.853194 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:28.853556 (+ 362us) server_negotiation.cc:407] Connection header received
0812 14:22:28.856008 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:28.856018 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:28.856177 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:28.856350 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:28.858322 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:28.860439 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:28.862486 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:28.863187 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:28.863370 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:28.868209 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:28.868215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:28.868216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:28.868231 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:28.870569 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:28.871151 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:28.871155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:28.871156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:28.871276 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:28.871590 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:28.871591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:28.871591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:28.871970 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:28.872198 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:28.872511 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 14:22:28.872724 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":105,"threads_started":1}
I20260812 14:22:28.874868 5824 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:28.875455 5824 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:28.879853 5824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e56fa4cd0b744a629e90e7785426ccf8: Generated new TSK 0
I20260812 14:22:28.880072 5824 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:22:28.912539 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:37435
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:29.011647 5842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:29.011886 5842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:29.011919 5842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:29.013339 5842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:29.013406 5842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:29.013437 5842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:22:29.015405 5842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:37435
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:29.015745 5842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:29.016027 5842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:29.023550 5851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:29.023533 5849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:29.024006 5842 server_base.cc:1034] running on GCE node
W20260812 14:22:29.024117 5848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:29.024525 5842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:29.024801 5842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:29.025979 5842 hybrid_clock.cc:648] HybridClock initialized: now 1786544549025966 us; error 32 us; skew 500 ppm
Aug 12 14:22:29 dist-test-slave-8w9v krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544549, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:29.029290 5842 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:22:29.029911 5842 webserver.cc:460] Webserver started at http://127.5.148.1:40855/ using document root <none> and password file <none>
I20260812 14:22:29.030174 5842 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:29.030244 5842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:29.030387 5842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:29.031739 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "abc657db38d446409c0a2d831405ca19"
format_stamp: "Formatted at 2026-08-12 14:22:29 on dist-test-slave-8w9v"
server_key: "8d663ecee4c2f564fddd183721397116"
server_key_iv: "02213e9347e7f27b3fc8c095414fa7ce"
server_key_version: "encryptionkey@0"
I20260812 14:22:29.032245 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "abc657db38d446409c0a2d831405ca19"
format_stamp: "Formatted at 2026-08-12 14:22:29 on dist-test-slave-8w9v"
server_key: "8d663ecee4c2f564fddd183721397116"
server_key_iv: "02213e9347e7f27b3fc8c095414fa7ce"
server_key_version: "encryptionkey@0"
I20260812 14:22:29.033972 5842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:29.034931 5861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:29.035099 5842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:29.035197 5842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "abc657db38d446409c0a2d831405ca19"
format_stamp: "Formatted at 2026-08-12 14:22:29 on dist-test-slave-8w9v"
server_key: "8d663ecee4c2f564fddd183721397116"
server_key_iv: "02213e9347e7f27b3fc8c095414fa7ce"
server_key_version: "encryptionkey@0"
I20260812 14:22:29.035272 5842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:29.055301 5842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:29.058020 5842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:29.058197 5842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:29.058566 5842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:29.059011 5842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:29.059098 5842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:29.059124 5842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:29.059175 5842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:29.068267 5842 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:38621
I20260812 14:22:29.068367 5986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:38621 every 8 connection(s)
I20260812 14:22:29.068707 5842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:22:29.069741 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 5842
I20260812 14:22:29.069841 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:22:29.070034 5712 external_mini_cluster.cc:1428] Setting key a74c14e4cee8df4ed7f7321d0b135b3c
Aug 12 14:22:29 dist-test-slave-8w9v krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544549, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
I20260812 14:22:29.081413 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:29.069758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:44597 (local address 127.5.148.62:37435)
0812 14:22:29.069842 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:29.069844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:29.070124 (+ 280us) server_negotiation.cc:407] Connection header received
0812 14:22:29.070767 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:29.070768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:29.070800 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:29.070851 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:29.071986 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.072844 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.074486 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.074699 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.074770 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:29.077828 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:29.077830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:29.077831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:29.077840 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:29.079517 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:29.080192 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:29.080194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:29.080194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:29.080262 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:29.080608 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:29.080609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:29.080609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:29.080729 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:29.080792 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:29.081185 (+ 393us) server_negotiation.cc:299] Negotiation successful
0812 14:22:29.081322 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 14:22:29.081682 5990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:29.069720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37435 (local address 127.5.148.1:44597)
0812 14:22:29.069953 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:29.070014 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:29.070653 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:29.070949 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:29.070953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:29.071643 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:29.071869 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:29.071874 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.072967 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.072968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:29.074365 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:29.074368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.074806 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.074808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:29.074977 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:29.075571 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:29.075582 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:29.077698 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:29.079617 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:29.079627 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:29.079631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:29.079968 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:29.080342 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:29.080344 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:29.080344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:29.080527 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:29.080807 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:29.080815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:29.081012 (+ 197us) client_negotiation.cc:769] Sending connection context
0812 14:22:29.081178 (+ 166us) client_negotiation.cc:240] Negotiation successful
0812 14:22:29.081400 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":121,"threads_started":1}
I20260812 14:22:29.082417 5988 heartbeater.cc:347] Connected to a master server at 127.5.148.62:37435
I20260812 14:22:29.082520 5988 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:29.082702 5988 heartbeater.cc:511] Master 127.5.148.62:37435 requested a full tablet report, sending...
I20260812 14:22:29.083626 5770 ts_manager.cc:194] Registered new tserver with Master: abc657db38d446409c0a2d831405ca19 (127.5.148.1:38621)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
I20260812 14:22:29.084542 5770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:44597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:22:29.102833 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:37435
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:29.193459 5994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:29.193692 5994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:29.193722 5994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:29.194979 5994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:29.195035 5994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:29.195066 5994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:22:29.196702 5994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:37435
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:29.197003 5994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:29.197283 5994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:29.202965 5999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:29.202929 6000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:29.202956 6002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:29.203351 5994 server_base.cc:1034] running on GCE node
I20260812 14:22:29.203599 5994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:29.204922 5994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:29.206130 5994 hybrid_clock.cc:648] HybridClock initialized: now 1786544549206108 us; error 42 us; skew 500 ppm
Aug 12 14:22:29 dist-test-slave-8w9v krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544549, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:29.209103 5994 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:22:29.209545 5994 webserver.cc:460] Webserver started at http://127.5.148.2:38663/ using document root <none> and password file <none>
I20260812 14:22:29.209757 5994 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:29.209798 5994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:29.209903 5994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:29.211052 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "055ebd65a973480d8b6ed383e8b7869f"
format_stamp: "Formatted at 2026-08-12 14:22:29 on dist-test-slave-8w9v"
server_key: "e6ccc986f91bb33fad6deb1858f1115b"
server_key_iv: "2768a6b837ea144b33385b9e6e97366f"
server_key_version: "encryptionkey@0"
I20260812 14:22:29.211387 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "055ebd65a973480d8b6ed383e8b7869f"
format_stamp: "Formatted at 2026-08-12 14:22:29 on dist-test-slave-8w9v"
server_key: "e6ccc986f91bb33fad6deb1858f1115b"
server_key_iv: "2768a6b837ea144b33385b9e6e97366f"
server_key_version: "encryptionkey@0"
I20260812 14:22:29.213126 5994 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 14:22:29.214094 6012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:29.214342 5994 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:29.214440 5994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "055ebd65a973480d8b6ed383e8b7869f"
format_stamp: "Formatted at 2026-08-12 14:22:29 on dist-test-slave-8w9v"
server_key: "e6ccc986f91bb33fad6deb1858f1115b"
server_key_iv: "2768a6b837ea144b33385b9e6e97366f"
server_key_version: "encryptionkey@0"
I20260812 14:22:29.214543 5994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:29.229830 5994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:29.231886 5994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:29.232053 5994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:29.232358 5994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:29.232697 5994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:29.232753 5994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:29.232800 5994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:29.232825 5994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:29.240665 5994 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:37349
I20260812 14:22:29.240706 6148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:37349 every 8 connection(s)
I20260812 14:22:29.241077 5994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:22:29 dist-test-slave-8w9v krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544549, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:29.249425 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 5994
I20260812 14:22:29.249538 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:22:29.249725 5712 external_mini_cluster.cc:1428] Setting key cce6e3acd33199158747c13272db3b71
I20260812 14:22:29.251281 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:29.241828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:49555 (local address 127.5.148.62:37435)
0812 14:22:29.241875 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:29.241883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:29.242148 (+ 265us) server_negotiation.cc:407] Connection header received
0812 14:22:29.242723 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:29.242725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:29.242738 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:29.242796 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:29.243851 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.244456 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.245552 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.245750 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.245838 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:29.248180 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:29.248182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:29.248182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:29.248198 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:29.249604 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:29.250192 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:29.250193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:29.250193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:29.250250 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:29.250569 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:29.250570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:29.250571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:29.250701 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:29.250785 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:29.251090 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 14:22:29.251186 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:22:29.251557 6151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:29.241823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37435 (local address 127.5.148.2:49555)
0812 14:22:29.242012 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:29.242053 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:29.242601 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:29.242879 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:29.242884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:29.243528 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:29.243742 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:29.243747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.244567 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.244568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:29.245455 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:29.245457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.245851 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.245852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:29.245992 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:29.246476 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:29.246484 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:29.248042 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:29.249701 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:29.249709 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:29.249711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:29.250070 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:29.250330 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:29.250331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:29.250331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:29.250508 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:29.250775 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:29.250784 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:29.250948 (+ 164us) client_negotiation.cc:769] Sending connection context
0812 14:22:29.251068 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 14:22:29.251248 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
I20260812 14:22:29.252112 6149 heartbeater.cc:347] Connected to a master server at 127.5.148.62:37435
I20260812 14:22:29.252241 6149 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:29.252454 6149 heartbeater.cc:511] Master 127.5.148.62:37435 requested a full tablet report, sending...
I20260812 14:22:29.252938 5770 ts_manager.cc:194] Registered new tserver with Master: 055ebd65a973480d8b6ed383e8b7869f (127.5.148.2:37349)
I20260812 14:22:29.253326 5770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:49555
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:22:29.279767 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:37435
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:29.373521 6155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:29.373756 6155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:29.373790 6155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:29.374972 6155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:29.375010 6155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:29.375038 6155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:22:29.376680 6155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:37435
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:29.376914 6155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:29.377107 6155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:29.382573 6161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:29.382586 6160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:29.382678 6164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:29.382956 6155 server_base.cc:1034] running on GCE node
I20260812 14:22:29.383177 6155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:29.383476 6155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:29.384635 6155 hybrid_clock.cc:648] HybridClock initialized: now 1786544549384614 us; error 44 us; skew 500 ppm
Aug 12 14:22:29 dist-test-slave-8w9v krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544549, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:29.387343 6155 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:22:29.387758 6155 webserver.cc:460] Webserver started at http://127.5.148.3:43673/ using document root <none> and password file <none>
I20260812 14:22:29.387970 6155 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:29.388019 6155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:29.388167 6155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:29.389307 6155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "69cf599680e84558ad8be7b5bb3290f9"
format_stamp: "Formatted at 2026-08-12 14:22:29 on dist-test-slave-8w9v"
server_key: "fd450dcce112eaab5718a6d550433491"
server_key_iv: "607a3012ddddb3b45e1811654192594d"
server_key_version: "encryptionkey@0"
I20260812 14:22:29.389645 6155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "69cf599680e84558ad8be7b5bb3290f9"
format_stamp: "Formatted at 2026-08-12 14:22:29 on dist-test-slave-8w9v"
server_key: "fd450dcce112eaab5718a6d550433491"
server_key_iv: "607a3012ddddb3b45e1811654192594d"
server_key_version: "encryptionkey@0"
I20260812 14:22:29.391054 6155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:22:29.391832 6172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:29.392028 6155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:29.392161 6155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "69cf599680e84558ad8be7b5bb3290f9"
format_stamp: "Formatted at 2026-08-12 14:22:29 on dist-test-slave-8w9v"
server_key: "fd450dcce112eaab5718a6d550433491"
server_key_iv: "607a3012ddddb3b45e1811654192594d"
server_key_version: "encryptionkey@0"
I20260812 14:22:29.392242 6155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:29.403223 6155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:29.405287 6155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:29.405411 6155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:29.405608 6155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:29.405949 6155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:29.405987 6155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:29.406010 6155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:29.406067 6155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:29.413472 6155 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:46109
I20260812 14:22:29.413580 6301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:46109 every 8 connection(s)
I20260812 14:22:29.414021 6155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:22:29.416169 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 6155
I20260812 14:22:29.416275 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:22:29.416440 5712 external_mini_cluster.cc:1428] Setting key d76f27e6cb38c0817d328cff7a691ebb
Aug 12 14:22:29 dist-test-slave-8w9v krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544549, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:29.424657 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:29.414874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:40301 (local address 127.5.148.62:37435)
0812 14:22:29.414931 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:29.414934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:29.415221 (+ 287us) server_negotiation.cc:407] Connection header received
0812 14:22:29.415732 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:29.415733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:29.415745 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:29.415809 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:29.417206 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.417960 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.419118 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.419316 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.419375 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:29.421806 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:29.421808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:29.421808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:29.421816 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:29.423315 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:29.423741 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:29.423742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:29.423743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:29.423798 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:29.424047 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:29.424047 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:29.424047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:29.424209 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:29.424263 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:29.424515 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 14:22:29.424586 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:22:29.424916 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:29.414809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37435 (local address 127.5.148.3:40301)
0812 14:22:29.415050 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:29.415118 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:29.415640 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:29.415872 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:29.415875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:29.416727 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:29.417107 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:29.417114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.418033 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.418033 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:29.419026 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:29.419029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.419398 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.419398 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:29.419515 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:29.420025 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:29.420035 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:29.421701 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:29.423413 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:29.423422 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:29.423424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:29.423668 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:29.423867 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:29.423867 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:29.423868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:29.423989 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:29.424255 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:29.424260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:29.424396 (+ 136us) client_negotiation.cc:769] Sending connection context
0812 14:22:29.424487 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 14:22:29.424625 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":114,"threads_started":1}
I20260812 14:22:29.425588 6302 heartbeater.cc:347] Connected to a master server at 127.5.148.62:37435
I20260812 14:22:29.425707 6302 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:29.425912 6302 heartbeater.cc:511] Master 127.5.148.62:37435 requested a full tablet report, sending...
I20260812 14:22:29.426359 5770 ts_manager.cc:194] Registered new tserver with Master: 69cf599680e84558ad8be7b5bb3290f9 (127.5.148.3:46109)
I20260812 14:22:29.426757 5770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:40301
I20260812 14:22:29.428918 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 14:22:29 dist-test-slave-8w9v krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544549, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 14:22:29 dist-test-slave-8w9v krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544549, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.148.1@KRBTEST.COM
I20260812 14:22:29.455806 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:29.447150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.5.148.1:38621)
0812 14:22:29.447382 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:29.447388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:29.447452 (+ 64us) server_negotiation.cc:407] Connection header received
0812 14:22:29.447505 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:29.447508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:29.447524 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:29.447599 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:29.448288 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.449113 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.449888 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.450227 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.450283 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:29.452905 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:29.452919 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:29.452920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:29.452929 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:29.454650 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:29.455015 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:29.455018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:29.455018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:29.455077 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:29.455249 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:29.455249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:29.455249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:29.455397 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:29.455539 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:29.455597 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 14:22:29.455668 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":140,"threads_started":1}
W20260812 14:22:29.456442 5890 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51532
Aug 12 14:22:29 dist-test-slave-8w9v krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 14:22:29 dist-test-slave-8w9v krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.1@KRBTEST.COM
I20260812 14:22:29.482333 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:29.474937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51534 (local address 127.5.148.1:38621)
0812 14:22:29.475009 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:29.475011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:29.475074 (+ 63us) server_negotiation.cc:407] Connection header received
0812 14:22:29.475142 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:29.475143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:29.475157 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:29.475212 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:29.475812 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.476394 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.477183 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:29.477383 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:29.477457 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:29.479866 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:29.479868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:29.479868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:29.479876 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:29.481384 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:29.481703 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:29.481703 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:29.481704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:29.481756 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:29.481918 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:29.481919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:29.481920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:29.482012 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:29.482102 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:29.482175 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 14:22:29.482240 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 14:22:29.482967 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 5842
I20260812 14:22:29.492259 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 5994
I20260812 14:22:29.502704 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 6155
I20260812 14:22:29.510169 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 5736
2026-08-12T14:22:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2212 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:22:29 dist-test-slave-8w9v krb5kdc[6319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:22:29 dist-test-slave-8w9v krb5kdc[6319](info): set up 2 sockets
Aug 12 14:22:29 dist-test-slave-8w9v krb5kdc[6319](info): commencing operation
krb5kdc: starting...
W20260812 14:22:30.650548 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:22:30 dist-test-slave-8w9v krb5kdc[6319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:22:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:22:30.784046 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:40911
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:40911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:30.870662 6334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:30.870939 6334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:30.870980 6334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:30.872195 6334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:30.872283 6334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:30.872306 6334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:30.872327 6334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:30.872359 6334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:22:30.874230 6334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:40911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:40911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:30.874634 6334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:30.874895 6334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:30.880209 6339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:30.880296 6334 server_base.cc:1034] running on GCE node
W20260812 14:22:30.880244 6342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:30.880216 6340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:30.880759 6334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:30.881078 6334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:30.882232 6334 hybrid_clock.cc:648] HybridClock initialized: now 1786544550882219 us; error 31 us; skew 500 ppm
Aug 12 14:22:30 dist-test-slave-8w9v krb5kdc[6319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544550, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:30.884950 6334 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:22:30.885460 6334 webserver.cc:460] Webserver started at http://127.5.148.62:36799/ using document root <none> and password file <none>
I20260812 14:22:30.885699 6334 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:30.885744 6334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:30.885902 6334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:30.887032 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "f5944c957bd5470ca4887d5666405716"
format_stamp: "Formatted at 2026-08-12 14:22:30 on dist-test-slave-8w9v"
server_key: "be33cc31d53b462504b7170ab9b7a5bf"
server_key_iv: "224a80fd64de188819c5cfa24d064f6f"
server_key_version: "encryptionkey@0"
I20260812 14:22:30.887416 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "f5944c957bd5470ca4887d5666405716"
format_stamp: "Formatted at 2026-08-12 14:22:30 on dist-test-slave-8w9v"
server_key: "be33cc31d53b462504b7170ab9b7a5bf"
server_key_iv: "224a80fd64de188819c5cfa24d064f6f"
server_key_version: "encryptionkey@0"
I20260812 14:22:30.888931 6334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:22:30.889801 6350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:30.890002 6334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:30.890115 6334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "f5944c957bd5470ca4887d5666405716"
format_stamp: "Formatted at 2026-08-12 14:22:30 on dist-test-slave-8w9v"
server_key: "be33cc31d53b462504b7170ab9b7a5bf"
server_key_iv: "224a80fd64de188819c5cfa24d064f6f"
server_key_version: "encryptionkey@0"
I20260812 14:22:30.890202 6334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:30.900377 6334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:30.902252 6334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:30.902421 6334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:30.906714 6334 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:40911
I20260812 14:22:30.906749 6410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:40911 every 8 connection(s)
I20260812 14:22:30.907130 6334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:30.908105 6411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:30.910010 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 6334
I20260812 14:22:30.910120 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:30.910326 5712 external_mini_cluster.cc:1428] Setting key 9419e61bff116c0f2e9d3d20939d8f95
I20260812 14:22:30.910813 6411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716: Bootstrap starting.
I20260812 14:22:30.911538 6411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:30.911876 6411 log.cc:824] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:30.912706 6411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716: No bootstrap required, opened a new log
I20260812 14:22:30.915359 6411 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5944c957bd5470ca4887d5666405716" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40911 } }
I20260812 14:22:30.915537 6411 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:30.915573 6411 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5944c957bd5470ca4887d5666405716, State: Initialized, Role: FOLLOWER
I20260812 14:22:30.915702 6411 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5944c957bd5470ca4887d5666405716" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40911 } }
I20260812 14:22:30.915805 6411 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:30.915838 6411 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:30.915892 6411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:30.916714 6411 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5944c957bd5470ca4887d5666405716" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40911 } }
I20260812 14:22:30.916816 6411 leader_election.cc:302] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5944c957bd5470ca4887d5666405716; no voters:
I20260812 14:22:30.917032 6411 leader_election.cc:288] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:30.917086 6418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:30.917207 6418 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [term 1 LEADER]: Becoming Leader. State: Replica: f5944c957bd5470ca4887d5666405716, State: Running, Role: LEADER
I20260812 14:22:30.917312 6411 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:30.917299 6418 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5944c957bd5470ca4887d5666405716" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40911 } }
I20260812 14:22:30.917649 6418 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5944c957bd5470ca4887d5666405716. Latest consensus state: current_term: 1 leader_uuid: "f5944c957bd5470ca4887d5666405716" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5944c957bd5470ca4887d5666405716" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40911 } } }
I20260812 14:22:30.917729 6418 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:30.917994 6424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 12 14:22:30 dist-test-slave-8w9v krb5kdc[6319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:30.919099 6424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:22:30.919445 6419 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5944c957bd5470ca4887d5666405716" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5944c957bd5470ca4887d5666405716" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40911 } } }
I20260812 14:22:30.919667 6419 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:30.921738 6424 catalog_manager.cc:1347] Generated new cluster ID: 7a0f7b859f934c7992a48c8e527aadfa
I20260812 14:22:30.921808 6424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:22:30.922986 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:30.910934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37062 (local address 127.5.148.62:40911)
0812 14:22:30.911139 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:30.911149 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:30.911187 (+ 38us) server_negotiation.cc:407] Connection header received
0812 14:22:30.911815 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:30.911824 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:30.911888 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:30.912041 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:30.912853 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:30.914259 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:30.915060 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:30.915694 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:30.915810 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:30.919117 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:30.919123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:30.919123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:30.919136 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:30.921254 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:30.921739 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:30.921742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:30.921742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:30.921883 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:30.922118 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:30.922119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:30.922119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:30.922370 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:30.922545 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:30.922655 (+ 110us) server_negotiation.cc:299] Negotiation successful
0812 14:22:30.922800 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260812 14:22:30.947558 6424 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:30.948175 6424 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:30.960402 6424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5944c957bd5470ca4887d5666405716: Generated new TSK 0
I20260812 14:22:30.960655 6424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:22:30.998690 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:40911
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:31.085505 6443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:31.085738 6443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:31.085767 6443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:31.087059 6443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:31.087113 6443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:31.087143 6443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:22:31.088845 6443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:40911
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:31.089150 6443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:31.089421 6443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:31.094954 6450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:31.094941 6453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:31.094954 6451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:31.095332 6443 server_base.cc:1034] running on GCE node
I20260812 14:22:31.095592 6443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:31.095840 6443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:31.096989 6443 hybrid_clock.cc:648] HybridClock initialized: now 1786544551096975 us; error 43 us; skew 500 ppm
Aug 12 14:22:31 dist-test-slave-8w9v krb5kdc[6319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544551, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:31.100073 6443 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:22:31.100594 6443 webserver.cc:460] Webserver started at http://127.5.148.1:35607/ using document root <none> and password file <none>
I20260812 14:22:31.100858 6443 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:31.100932 6443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:31.101084 6443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:31.102198 6443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "5e7ad9fccee148769f201c48500a28ae"
format_stamp: "Formatted at 2026-08-12 14:22:31 on dist-test-slave-8w9v"
server_key: "6d7430811d499334a9480688b2df8cd5"
server_key_iv: "de84706ef7d449787f0a286e7d603871"
server_key_version: "encryptionkey@0"
I20260812 14:22:31.102588 6443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "5e7ad9fccee148769f201c48500a28ae"
format_stamp: "Formatted at 2026-08-12 14:22:31 on dist-test-slave-8w9v"
server_key: "6d7430811d499334a9480688b2df8cd5"
server_key_iv: "de84706ef7d449787f0a286e7d603871"
server_key_version: "encryptionkey@0"
I20260812 14:22:31.103986 6443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260812 14:22:31.104911 6461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:31.105098 6443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:22:31.105204 6443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "5e7ad9fccee148769f201c48500a28ae"
format_stamp: "Formatted at 2026-08-12 14:22:31 on dist-test-slave-8w9v"
server_key: "6d7430811d499334a9480688b2df8cd5"
server_key_iv: "de84706ef7d449787f0a286e7d603871"
server_key_version: "encryptionkey@0"
I20260812 14:22:31.105296 6443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:31.119695 6443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:31.122375 6443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:31.122565 6443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:31.122851 6443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:31.123205 6443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:31.123276 6443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:31.123317 6443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:31.123364 6443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:31.130541 6443 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:36299
I20260812 14:22:31.130586 6600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:36299 every 8 connection(s)
I20260812 14:22:31.130949 6443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:22:31.134922 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 6443
I20260812 14:22:31.135010 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:22:31.135183 5712 external_mini_cluster.cc:1428] Setting key 475e1aab3763b91e83622ca298f5a6ff
Aug 12 14:22:31 dist-test-slave-8w9v krb5kdc[6319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544551, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:31.142513 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.131832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:46161 (local address 127.5.148.62:40911)
0812 14:22:31.131922 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:31.131925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:31.132140 (+ 215us) server_negotiation.cc:407] Connection header received
0812 14:22:31.133404 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:31.133406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:31.133421 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:31.133472 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:31.134170 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.134688 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.136091 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.136357 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.136430 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.139125 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:31.139127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:31.139127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:31.139135 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:31.140685 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:31.141290 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:31.141291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:31.141292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:31.141392 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:31.141744 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:31.141745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:31.141745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:31.141870 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:31.141939 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:31.142319 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 14:22:31.142434 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:22:31.142814 6603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.131782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:40911 (local address 127.5.148.1:46161)
0812 14:22:31.131975 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:31.132028 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:31.133263 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:31.133567 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:31.133572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:31.133864 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:31.134068 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.134072 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.134796 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.134798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.135979 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.135982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.136452 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.136453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.136597 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.137172 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:31.137181 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:31.138984 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:31.140780 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:31.140789 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:31.140792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:31.141158 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:31.141484 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:31.141486 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:31.141486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:31.141651 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:31.141951 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:31.141959 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:31.142159 (+ 200us) client_negotiation.cc:769] Sending connection context
0812 14:22:31.142297 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 14:22:31.142507 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260812 14:22:31.143453 6601 heartbeater.cc:347] Connected to a master server at 127.5.148.62:40911
I20260812 14:22:31.143580 6601 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:31.143796 6601 heartbeater.cc:511] Master 127.5.148.62:40911 requested a full tablet report, sending...
I20260812 14:22:31.144402 6369 ts_manager.cc:194] Registered new tserver with Master: 5e7ad9fccee148769f201c48500a28ae (127.5.148.1:36299)
I20260812 14:22:31.145069 6369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:46161
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:22:31.165521 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:40911
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:31.257200 6606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:31.257421 6606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:31.257447 6606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:31.258540 6606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:31.258579 6606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:31.258605 6606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:22:31.260043 6606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:40911
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:31.260319 6606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:31.260524 6606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:31.265990 6613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:31.265976 6606 server_base.cc:1034] running on GCE node
W20260812 14:22:31.265959 6616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:31.265952 6611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:31.266347 6606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:31.266613 6606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:31.267767 6606 hybrid_clock.cc:648] HybridClock initialized: now 1786544551267756 us; error 28 us; skew 500 ppm
Aug 12 14:22:31 dist-test-slave-8w9v krb5kdc[6319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544551, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:31.275780 6606 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:22:31.276367 6606 webserver.cc:460] Webserver started at http://127.5.148.2:34509/ using document root <none> and password file <none>
I20260812 14:22:31.276608 6606 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:31.276678 6606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:31.276819 6606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:31.277959 6606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "77c1d135370845e8948953374d2355d2"
format_stamp: "Formatted at 2026-08-12 14:22:31 on dist-test-slave-8w9v"
server_key: "007d7f777551d67792ec273971ae189a"
server_key_iv: "0278d35ce7b7216df7a9c6c04ffe684d"
server_key_version: "encryptionkey@0"
I20260812 14:22:31.278282 6606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "77c1d135370845e8948953374d2355d2"
format_stamp: "Formatted at 2026-08-12 14:22:31 on dist-test-slave-8w9v"
server_key: "007d7f777551d67792ec273971ae189a"
server_key_iv: "0278d35ce7b7216df7a9c6c04ffe684d"
server_key_version: "encryptionkey@0"
I20260812 14:22:31.279656 6606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:22:31.280438 6624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:31.280642 6606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:31.280722 6606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "77c1d135370845e8948953374d2355d2"
format_stamp: "Formatted at 2026-08-12 14:22:31 on dist-test-slave-8w9v"
server_key: "007d7f777551d67792ec273971ae189a"
server_key_iv: "0278d35ce7b7216df7a9c6c04ffe684d"
server_key_version: "encryptionkey@0"
I20260812 14:22:31.280834 6606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:31.293077 6606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:31.295377 6606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:31.295558 6606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:31.295847 6606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:31.296341 6606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:31.296401 6606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:31.296453 6606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:31.296478 6606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:31.303778 6606 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:33093
I20260812 14:22:31.303820 6750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:33093 every 8 connection(s)
I20260812 14:22:31.304328 6606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:22:31 dist-test-slave-8w9v krb5kdc[6319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544551, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:31.312027 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 6606
I20260812 14:22:31.312155 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:22:31.312350 5712 external_mini_cluster.cc:1428] Setting key 2a57555d5f7bfc5db8c60d135b8432b0
I20260812 14:22:31.315166 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.305213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:33305 (local address 127.5.148.62:40911)
0812 14:22:31.305295 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:31.305297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:31.305629 (+ 332us) server_negotiation.cc:407] Connection header received
0812 14:22:31.306579 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:31.306581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:31.306593 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:31.306659 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:31.307272 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.307802 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.309014 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.309220 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.309405 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.311726 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:31.311728 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:31.311729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:31.311737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:31.313475 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:31.313987 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:31.313989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:31.313989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:31.314071 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:31.314421 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:31.314422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:31.314423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:31.314558 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:31.314646 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:31.314966 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 14:22:31.315068 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 14:22:31.315402 6755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.305198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:40911 (local address 127.5.148.2:33305)
0812 14:22:31.305435 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:31.305492 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:31.306479 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:31.306716 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:31.306719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:31.306988 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:31.307192 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.307196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.307902 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.307903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.308902 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.308905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.309413 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.309414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.309524 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.310094 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:31.310103 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:31.311608 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:31.313594 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:31.313604 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:31.313607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:31.313879 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:31.314175 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:31.314176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:31.314176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:31.314333 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:31.314636 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:31.314645 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:31.314817 (+ 172us) client_negotiation.cc:769] Sending connection context
0812 14:22:31.314943 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 14:22:31.315130 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
I20260812 14:22:31.315939 6751 heartbeater.cc:347] Connected to a master server at 127.5.148.62:40911
I20260812 14:22:31.316089 6751 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:31.316344 6751 heartbeater.cc:511] Master 127.5.148.62:40911 requested a full tablet report, sending...
I20260812 14:22:31.316840 6369 ts_manager.cc:194] Registered new tserver with Master: 77c1d135370845e8948953374d2355d2 (127.5.148.2:33093)
I20260812 14:22:31.317307 6369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:33305
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:22:31.342744 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:40911
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:31.435344 6758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:31.435565 6758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:31.435592 6758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:31.436914 6758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:31.436966 6758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:31.436995 6758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:22:31.438674 6758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:40911
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:31.438946 6758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:31.439224 6758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:31.444799 6767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:31.444829 6764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:31.444803 6765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:31.445068 6758 server_base.cc:1034] running on GCE node
I20260812 14:22:31.445290 6758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:31.445528 6758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:31.446676 6758 hybrid_clock.cc:648] HybridClock initialized: now 1786544551446647 us; error 53 us; skew 500 ppm
Aug 12 14:22:31 dist-test-slave-8w9v krb5kdc[6319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544551, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:31.449386 6758 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:22:31.449801 6758 webserver.cc:460] Webserver started at http://127.5.148.3:36761/ using document root <none> and password file <none>
I20260812 14:22:31.450029 6758 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:31.450073 6758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:31.450171 6758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:31.451262 6758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "deb2a627e2f44687b69ec5c600ffd419"
format_stamp: "Formatted at 2026-08-12 14:22:31 on dist-test-slave-8w9v"
server_key: "e698a3103339e459543371661778388f"
server_key_iv: "9fd0fc2973043fd2a64f8679906af896"
server_key_version: "encryptionkey@0"
I20260812 14:22:31.451581 6758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "deb2a627e2f44687b69ec5c600ffd419"
format_stamp: "Formatted at 2026-08-12 14:22:31 on dist-test-slave-8w9v"
server_key: "e698a3103339e459543371661778388f"
server_key_iv: "9fd0fc2973043fd2a64f8679906af896"
server_key_version: "encryptionkey@0"
I20260812 14:22:31.453056 6758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:22:31.453820 6774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:31.454022 6758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:31.454118 6758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "deb2a627e2f44687b69ec5c600ffd419"
format_stamp: "Formatted at 2026-08-12 14:22:31 on dist-test-slave-8w9v"
server_key: "e698a3103339e459543371661778388f"
server_key_iv: "9fd0fc2973043fd2a64f8679906af896"
server_key_version: "encryptionkey@0"
I20260812 14:22:31.454181 6758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:31.480974 6758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:31.483752 6758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:31.483999 6758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:31.484426 6758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:31.484805 6758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:31.484874 6758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:31.484922 6758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:31.484951 6758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:31.492254 6758 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:38373
I20260812 14:22:31.492334 6914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:38373 every 8 connection(s)
I20260812 14:22:31.492722 6758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:22:31.499414 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 6758
I20260812 14:22:31.499511 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:22:31.499696 5712 external_mini_cluster.cc:1428] Setting key ccb2893a1913ce737e195b4c3d5212a5
Aug 12 14:22:31 dist-test-slave-8w9v krb5kdc[6319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544551, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:31.503343 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.493755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:49399 (local address 127.5.148.62:40911)
0812 14:22:31.493843 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:31.493845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:31.494092 (+ 247us) server_negotiation.cc:407] Connection header received
0812 14:22:31.495202 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:31.495203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:31.495219 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:31.495267 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:31.495974 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.496578 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.497670 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.497895 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.497978 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.500696 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:31.500698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:31.500698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:31.500708 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:31.501989 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:31.502398 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:31.502399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:31.502399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:31.502457 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:31.502688 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:31.502688 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:31.502689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:31.502798 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:31.502850 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:31.503206 (+ 356us) server_negotiation.cc:299] Negotiation successful
0812 14:22:31.503275 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 14:22:31.503600 6917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.493732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:40911 (local address 127.5.148.3:49399)
0812 14:22:31.493949 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:31.493991 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:31.495064 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:31.495375 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:31.495384 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:31.495656 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:31.495868 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.495873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.496687 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.496688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.497544 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.497547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.497989 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.497990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.498140 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.498708 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:31.498718 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:31.500543 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:31.502069 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:31.502076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:31.502078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:31.502320 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:31.502519 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:31.502519 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:31.502520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:31.502630 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:31.502925 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:31.502933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:31.503076 (+ 143us) client_negotiation.cc:769] Sending connection context
0812 14:22:31.503180 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 14:22:31.503337 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":102,"threads_started":1}
I20260812 14:22:31.504084 6915 heartbeater.cc:347] Connected to a master server at 127.5.148.62:40911
I20260812 14:22:31.504235 6915 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:31.504470 6915 heartbeater.cc:511] Master 127.5.148.62:40911 requested a full tablet report, sending...
I20260812 14:22:31.504876 6369 ts_manager.cc:194] Registered new tserver with Master: deb2a627e2f44687b69ec5c600ffd419 (127.5.148.3:38373)
I20260812 14:22:31.505275 6369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:49399
I20260812 14:22:31.511883 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:22:31.521163 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.514050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.5.148.62:40911)
0812 14:22:31.514118 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:31.514120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:31.514230 (+ 110us) server_negotiation.cc:407] Connection header received
0812 14:22:31.514276 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:31.514277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:31.514287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:31.514344 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:31.514993 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.515485 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.516348 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.516555 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.516610 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.518256 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:31.518258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:31.518258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:31.518265 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:31.519857 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:31.520341 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:31.520342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:31.520342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:31.520397 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:31.520654 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:31.520654 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:31.520655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:31.520802 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:31.520880 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:31.521024 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 14:22:31.521094 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 14:22:31.524559 6369 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:22:31.526417 6369 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:22:31.533964 6927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.529616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:36299 (local address 127.0.0.1:53880)
0812 14:22:31.530286 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:31.530296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:31.530366 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:31.530580 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:31.530581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:31.530582 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:31.530629 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.530631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.531678 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.531680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.532875 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.532878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.533645 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.533646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.533728 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.533743 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 14:22:31.533821 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 14:22:31.533875 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":1,"thread_start_us":105,"threads_started":1}
I20260812 14:22:31.533985 6929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.529766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.5.148.1:36299)
0812 14:22:31.530237 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:31.530241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:31.530422 (+ 181us) server_negotiation.cc:407] Connection header received
0812 14:22:31.530469 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:31.530472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:31.530487 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:31.530564 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:31.530774 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.531563 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.532971 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.533559 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.533631 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.533693 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:31.533856 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 14:22:31.533900 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":96,"threads_started":1}
I20260812 14:22:31.534598 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.529613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:38373 (local address 127.0.0.1:57128)
0812 14:22:31.529793 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:31.529837 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:31.530172 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:31.530440 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:31.530442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:31.530445 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:31.530518 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.530520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.532061 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.532062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.533353 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.533356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.534338 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.534339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.534405 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.534409 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:22:31.534483 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 14:22:31.534527 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":84,"thread_start_us":62,"threads_started":1}
I20260812 14:22:31.534621 6932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.529985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.5.148.3:38373)
0812 14:22:31.530177 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:31.530183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:31.530197 (+ 14us) server_negotiation.cc:407] Connection header received
0812 14:22:31.530290 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:31.530293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:31.530315 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:31.530629 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:31.530774 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.531920 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.533469 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.534248 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.534336 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.534396 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:31.534497 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 14:22:31.534538 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":116,"threads_started":1}
I20260812 14:22:31.534991 6522 tablet_service.cc:1467] Processing CreateTablet for tablet 086204a8f3b54b8c942c27d2c639143a (DEFAULT_TABLE table=test-table [id=85fb2fc131db44d4a1faa5f064d893a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:31.535339 6930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.529829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:33093 (local address 127.0.0.1:58102)
0812 14:22:31.530299 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:31.530307 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:31.530371 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:31.530720 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:31.530721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:31.530722 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:31.530749 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.530750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.532059 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.532061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.533353 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.533356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.535086 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.535087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.535149 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.535154 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:22:31.535214 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 14:22:31.535264 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":39,"thread_start_us":102,"threads_started":1}
I20260812 14:22:31.535386 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.529965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.5.148.2:33093)
0812 14:22:31.530151 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:31.530157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:31.530373 (+ 216us) server_negotiation.cc:407] Connection header received
0812 14:22:31.530463 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:31.530465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:31.530481 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:31.530564 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:31.530835 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.531924 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.533478 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.534953 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.535084 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.535144 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:31.535233 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 14:22:31.535282 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":110,"threads_started":1}
I20260812 14:22:31.535418 6522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086204a8f3b54b8c942c27d2c639143a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:31.535897 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 086204a8f3b54b8c942c27d2c639143a (DEFAULT_TABLE table=test-table [id=85fb2fc131db44d4a1faa5f064d893a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:31.536298 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086204a8f3b54b8c942c27d2c639143a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:31.536263 6677 tablet_service.cc:1467] Processing CreateTablet for tablet 086204a8f3b54b8c942c27d2c639143a (DEFAULT_TABLE table=test-table [id=85fb2fc131db44d4a1faa5f064d893a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:31.536602 6677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086204a8f3b54b8c942c27d2c639143a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:31.539337 6933 tablet_bootstrap.cc:492] T 086204a8f3b54b8c942c27d2c639143a P 5e7ad9fccee148769f201c48500a28ae: Bootstrap starting.
I20260812 14:22:31.539772 6934 tablet_bootstrap.cc:492] T 086204a8f3b54b8c942c27d2c639143a P 77c1d135370845e8948953374d2355d2: Bootstrap starting.
I20260812 14:22:31.540289 6935 tablet_bootstrap.cc:492] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419: Bootstrap starting.
I20260812 14:22:31.540386 6933 tablet_bootstrap.cc:654] T 086204a8f3b54b8c942c27d2c639143a P 5e7ad9fccee148769f201c48500a28ae: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:31.540755 6934 tablet_bootstrap.cc:654] T 086204a8f3b54b8c942c27d2c639143a P 77c1d135370845e8948953374d2355d2: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:31.541167 6935 tablet_bootstrap.cc:654] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:31.541227 6933 log.cc:824] T 086204a8f3b54b8c942c27d2c639143a P 5e7ad9fccee148769f201c48500a28ae: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:31.541435 6934 log.cc:824] T 086204a8f3b54b8c942c27d2c639143a P 77c1d135370845e8948953374d2355d2: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:31.541864 6935 log.cc:824] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:31.542284 6934 tablet_bootstrap.cc:492] T 086204a8f3b54b8c942c27d2c639143a P 77c1d135370845e8948953374d2355d2: No bootstrap required, opened a new log
I20260812 14:22:31.542385 6934 ts_tablet_manager.cc:1397] T 086204a8f3b54b8c942c27d2c639143a P 77c1d135370845e8948953374d2355d2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:22:31.542654 6935 tablet_bootstrap.cc:492] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419: No bootstrap required, opened a new log
I20260812 14:22:31.542713 6935 ts_tablet_manager.cc:1397] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:22:31.544660 6933 tablet_bootstrap.cc:492] T 086204a8f3b54b8c942c27d2c639143a P 5e7ad9fccee148769f201c48500a28ae: No bootstrap required, opened a new log
I20260812 14:22:31.544555 6935 raft_consensus.cc:348] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb2a627e2f44687b69ec5c600ffd419" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 38373 } } peers { permanent_uuid: "77c1d135370845e8948953374d2355d2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33093 } } peers { permanent_uuid: "5e7ad9fccee148769f201c48500a28ae" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36299 } }
I20260812 14:22:31.544741 6933 ts_tablet_manager.cc:1397] T 086204a8f3b54b8c942c27d2c639143a P 5e7ad9fccee148769f201c48500a28ae: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 14:22:31.544756 6935 raft_consensus.cc:374] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:31.544808 6935 raft_consensus.cc:729] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb2a627e2f44687b69ec5c600ffd419, State: Initialized, Role: FOLLOWER
I20260812 14:22:31.544752 6934 raft_consensus.cc:348] T 086204a8f3b54b8c942c27d2c639143a P 77c1d135370845e8948953374d2355d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb2a627e2f44687b69ec5c600ffd419" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 38373 } } peers { permanent_uuid: "77c1d135370845e8948953374d2355d2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33093 } } peers { permanent_uuid: "5e7ad9fccee148769f201c48500a28ae" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36299 } }
I20260812 14:22:31.544975 6935 consensus_queue.cc:260] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb2a627e2f44687b69ec5c600ffd419" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 38373 } } peers { permanent_uuid: "77c1d135370845e8948953374d2355d2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33093 } } peers { permanent_uuid: "5e7ad9fccee148769f201c48500a28ae" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36299 } }
I20260812 14:22:31.545030 6934 raft_consensus.cc:374] T 086204a8f3b54b8c942c27d2c639143a P 77c1d135370845e8948953374d2355d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:31.545080 6934 raft_consensus.cc:729] T 086204a8f3b54b8c942c27d2c639143a P 77c1d135370845e8948953374d2355d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c1d135370845e8948953374d2355d2, State: Initialized, Role: FOLLOWER
I20260812 14:22:31.545246 6934 consensus_queue.cc:260] T 086204a8f3b54b8c942c27d2c639143a P 77c1d135370845e8948953374d2355d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb2a627e2f44687b69ec5c600ffd419" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 38373 } } peers { permanent_uuid: "77c1d135370845e8948953374d2355d2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33093 } } peers { permanent_uuid: "5e7ad9fccee148769f201c48500a28ae" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36299 } }
I20260812 14:22:31.545285 6935 ts_tablet_manager.cc:1428] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 14:22:31.545529 6915 heartbeater.cc:503] Master 127.5.148.62:40911 was elected leader, sending a full tablet report...
I20260812 14:22:31.545629 6751 heartbeater.cc:503] Master 127.5.148.62:40911 was elected leader, sending a full tablet report...
I20260812 14:22:31.547430 6933 raft_consensus.cc:348] T 086204a8f3b54b8c942c27d2c639143a P 5e7ad9fccee148769f201c48500a28ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb2a627e2f44687b69ec5c600ffd419" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 38373 } } peers { permanent_uuid: "77c1d135370845e8948953374d2355d2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33093 } } peers { permanent_uuid: "5e7ad9fccee148769f201c48500a28ae" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36299 } }
I20260812 14:22:31.547600 6934 ts_tablet_manager.cc:1428] T 086204a8f3b54b8c942c27d2c639143a P 77c1d135370845e8948953374d2355d2: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260812 14:22:31.547648 6933 raft_consensus.cc:374] T 086204a8f3b54b8c942c27d2c639143a P 5e7ad9fccee148769f201c48500a28ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:31.547694 6933 raft_consensus.cc:729] T 086204a8f3b54b8c942c27d2c639143a P 5e7ad9fccee148769f201c48500a28ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7ad9fccee148769f201c48500a28ae, State: Initialized, Role: FOLLOWER
I20260812 14:22:31.547839 6933 consensus_queue.cc:260] T 086204a8f3b54b8c942c27d2c639143a P 5e7ad9fccee148769f201c48500a28ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb2a627e2f44687b69ec5c600ffd419" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 38373 } } peers { permanent_uuid: "77c1d135370845e8948953374d2355d2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33093 } } peers { permanent_uuid: "5e7ad9fccee148769f201c48500a28ae" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36299 } }
I20260812 14:22:31.548194 6933 ts_tablet_manager.cc:1428] T 086204a8f3b54b8c942c27d2c639143a P 5e7ad9fccee148769f201c48500a28ae: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 14:22:31.548214 6601 heartbeater.cc:503] Master 127.5.148.62:40911 was elected leader, sending a full tablet report...
W20260812 14:22:31.555042 6754 tablet.cc:2369] T 086204a8f3b54b8c942c27d2c639143a P 77c1d135370845e8948953374d2355d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:22:31.631865 6602 tablet.cc:2369] T 086204a8f3b54b8c942c27d2c639143a P 5e7ad9fccee148769f201c48500a28ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:22:31.743637 6916 tablet.cc:2369] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:22:31.776057 6939 raft_consensus.cc:482] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:22:31.776296 6939 raft_consensus.cc:504] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb2a627e2f44687b69ec5c600ffd419" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 38373 } } peers { permanent_uuid: "77c1d135370845e8948953374d2355d2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33093 } } peers { permanent_uuid: "5e7ad9fccee148769f201c48500a28ae" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36299 } }
I20260812 14:22:31.776949 6939 leader_election.cc:288] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77c1d135370845e8948953374d2355d2 (127.5.148.2:33093), 5e7ad9fccee148769f201c48500a28ae (127.5.148.1:36299)
I20260812 14:22:31.780978 6929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.777290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:47103 (local address 127.5.148.1:36299)
0812 14:22:31.777351 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:31.777353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:31.777603 (+ 250us) server_negotiation.cc:407] Connection header received
0812 14:22:31.777662 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:31.777663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:31.777678 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:31.777731 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:31.777878 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.778566 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.780004 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.780583 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.780658 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.780671 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:31.780864 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 14:22:31.780903 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:22:31.780979 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.777285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:52875 (local address 127.5.148.2:33093)
0812 14:22:31.777345 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:31.777348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:31.777360 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:22:31.777398 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:31.777403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:31.777415 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:31.777482 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:31.777692 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.778499 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.780002 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.780643 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.780716 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.780724 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:31.780864 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 14:22:31.780907 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 14:22:31.780994 6917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.777192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:33093 (local address 127.5.148.3:52875)
0812 14:22:31.777251 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:31.777264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:31.777312 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:31.777515 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:31.777517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:31.777519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:31.777605 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.777607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.778588 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.778589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.779880 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.779883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.780715 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.780716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.780786 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.780787 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:22:31.780842 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 14:22:31.780888 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":6}
I20260812 14:22:31.781002 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.777203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:36299 (local address 127.5.148.3:47103)
0812 14:22:31.777518 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:31.777528 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:31.777588 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:31.777749 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:31.777750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:31.777751 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:31.777805 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.777806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.778645 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.778646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.779880 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:31.779883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.780696 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.780697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:31.780778 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.780782 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:22:31.780842 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 14:22:31.780887 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":82,"thread_start_us":139,"threads_started":1}
I20260812 14:22:31.781494 6698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086204a8f3b54b8c942c27d2c639143a" candidate_uuid: "deb2a627e2f44687b69ec5c600ffd419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c1d135370845e8948953374d2355d2" is_pre_election: true
I20260812 14:22:31.781585 6545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086204a8f3b54b8c942c27d2c639143a" candidate_uuid: "deb2a627e2f44687b69ec5c600ffd419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7ad9fccee148769f201c48500a28ae" is_pre_election: true
I20260812 14:22:31.781750 6545 raft_consensus.cc:2454] T 086204a8f3b54b8c942c27d2c639143a P 5e7ad9fccee148769f201c48500a28ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate deb2a627e2f44687b69ec5c600ffd419 in term 0.
I20260812 14:22:31.781750 6698 raft_consensus.cc:2454] T 086204a8f3b54b8c942c27d2c639143a P 77c1d135370845e8948953374d2355d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate deb2a627e2f44687b69ec5c600ffd419 in term 0.
I20260812 14:22:31.782068 6778 leader_election.cc:302] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c1d135370845e8948953374d2355d2, deb2a627e2f44687b69ec5c600ffd419; no voters:
I20260812 14:22:31.782204 6939 raft_consensus.cc:2781] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:22:31.782270 6939 raft_consensus.cc:482] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:22:31.782320 6939 raft_consensus.cc:3037] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:31.783344 6939 raft_consensus.cc:504] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb2a627e2f44687b69ec5c600ffd419" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 38373 } } peers { permanent_uuid: "77c1d135370845e8948953374d2355d2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33093 } } peers { permanent_uuid: "5e7ad9fccee148769f201c48500a28ae" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36299 } }
I20260812 14:22:31.783519 6939 leader_election.cc:288] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [CANDIDATE]: Term 1 election: Requested vote from peers 77c1d135370845e8948953374d2355d2 (127.5.148.2:33093), 5e7ad9fccee148769f201c48500a28ae (127.5.148.1:36299)
I20260812 14:22:31.783711 6698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086204a8f3b54b8c942c27d2c639143a" candidate_uuid: "deb2a627e2f44687b69ec5c600ffd419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c1d135370845e8948953374d2355d2"
I20260812 14:22:31.783795 6698 raft_consensus.cc:3037] T 086204a8f3b54b8c942c27d2c639143a P 77c1d135370845e8948953374d2355d2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:31.784173 6545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086204a8f3b54b8c942c27d2c639143a" candidate_uuid: "deb2a627e2f44687b69ec5c600ffd419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7ad9fccee148769f201c48500a28ae"
I20260812 14:22:31.784269 6545 raft_consensus.cc:3037] T 086204a8f3b54b8c942c27d2c639143a P 5e7ad9fccee148769f201c48500a28ae [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:31.784787 6698 raft_consensus.cc:2454] T 086204a8f3b54b8c942c27d2c639143a P 77c1d135370845e8948953374d2355d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate deb2a627e2f44687b69ec5c600ffd419 in term 1.
I20260812 14:22:31.785040 6778 leader_election.cc:302] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c1d135370845e8948953374d2355d2, deb2a627e2f44687b69ec5c600ffd419; no voters:
I20260812 14:22:31.785184 6939 raft_consensus.cc:2781] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:31.785439 6939 raft_consensus.cc:686] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [term 1 LEADER]: Becoming Leader. State: Replica: deb2a627e2f44687b69ec5c600ffd419, State: Running, Role: LEADER
I20260812 14:22:31.785543 6545 raft_consensus.cc:2454] T 086204a8f3b54b8c942c27d2c639143a P 5e7ad9fccee148769f201c48500a28ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate deb2a627e2f44687b69ec5c600ffd419 in term 1.
I20260812 14:22:31.785564 6939 consensus_queue.cc:237] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb2a627e2f44687b69ec5c600ffd419" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 38373 } } peers { permanent_uuid: "77c1d135370845e8948953374d2355d2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33093 } } peers { permanent_uuid: "5e7ad9fccee148769f201c48500a28ae" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36299 } }
I20260812 14:22:31.786712 6368 catalog_manager.cc:5495] T 086204a8f3b54b8c942c27d2c639143a P deb2a627e2f44687b69ec5c600ffd419 reported cstate change: term changed from 0 to 1, leader changed from <none> to deb2a627e2f44687b69ec5c600ffd419 (127.5.148.3). New cstate: current_term: 1 leader_uuid: "deb2a627e2f44687b69ec5c600ffd419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb2a627e2f44687b69ec5c600ffd419" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 38373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77c1d135370845e8948953374d2355d2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e7ad9fccee148769f201c48500a28ae" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36299 } health_report { overall_health: UNKNOWN } } }
Aug 12 14:22:31 dist-test-slave-8w9v krb5kdc[6319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.1@KRBTEST.COM
I20260812 14:22:31.824990 6929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.817121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.5.148.1:36299)
0812 14:22:31.817187 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:31.817190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:31.817490 (+ 300us) server_negotiation.cc:407] Connection header received
0812 14:22:31.817623 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:31.817624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:31.817638 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:31.817707 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:31.818432 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.818928 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.819736 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.819942 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.820038 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.822442 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:31.822456 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:31.822456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:31.822463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:31.823940 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:31.824271 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:31.824275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:31.824275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:31.824341 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:31.824552 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:31.824553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:31.824553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:31.824690 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:31.824796 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:31.824856 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 14:22:31.824920 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Aug 12 14:22:31 dist-test-slave-8w9v krb5kdc[6319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544551, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 14:22:31 dist-test-slave-8w9v krb5kdc[6319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544551, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.148.1@KRBTEST.COM
I20260812 14:22:31.850788 6929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.843500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.5.148.1:36299)
0812 14:22:31.843598 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:31.843600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:31.843645 (+ 45us) server_negotiation.cc:407] Connection header received
0812 14:22:31.843695 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:31.843696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:31.843706 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:31.843760 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:31.844404 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.844913 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.845613 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.845820 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.845873 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.848258 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:31.848259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:31.848259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:31.848266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:31.849621 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:31.849981 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:31.849982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:31.849983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:31.850039 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:31.850208 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:31.850209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:31.850209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:31.850312 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:31.850390 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:31.850568 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 14:22:31.850640 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Aug 12 14:22:31 dist-test-slave-8w9v krb5kdc[6319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 14:22:31 dist-test-slave-8w9v krb5kdc[6319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.1@KRBTEST.COM
I20260812 14:22:31.875451 6929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:31.868442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.5.148.1:36299)
0812 14:22:31.868519 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:31.868521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:31.868582 (+ 61us) server_negotiation.cc:407] Connection header received
0812 14:22:31.868653 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:31.868654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:31.868664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:31.868720 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:31.869310 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.869781 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.870526 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:31.870702 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:31.870753 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:31.873132 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:31.873133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:31.873133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:31.873140 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:31.874390 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:31.874740 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:31.874742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:31.874742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:31.874819 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:31.875011 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:31.875012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:31.875012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:31.875114 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:31.875192 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:31.875280 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 14:22:31.875344 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260812 14:22:31.876477 6498 tablet_service.cc:2908] Rejecting scan request for tablet 086204a8f3b54b8c942c27d2c639143a: Uninitialized: clean time has not yet been initialized
I20260812 14:22:31.877592 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 6443
I20260812 14:22:31.886718 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 6606
I20260812 14:22:31.897518 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 6758
I20260812 14:22:31.905285 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 6334
2026-08-12T14:22:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2394 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:22:31 dist-test-slave-8w9v krb5kdc[6961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:22:31 dist-test-slave-8w9v krb5kdc[6961](info): set up 2 sockets
Aug 12 14:22:31 dist-test-slave-8w9v krb5kdc[6961](info): commencing operation
krb5kdc: starting...
W20260812 14:22:33.052433 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:22:33 dist-test-slave-8w9v krb5kdc[6961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:22:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:22:33.187090 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:38989
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:38989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:33.273146 6976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:33.273438 6976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:33.273483 6976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:33.274689 6976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:33.274775 6976 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 14:22:33.274798 6976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:33.274820 6976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:33.274844 6976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:33.274874 6976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:22:33.276539 6976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:38989
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:38989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:33.276840 6976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:33.277094 6976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:33.282652 6984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:33.282792 6986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:33.282636 6976 server_base.cc:1034] running on GCE node
W20260812 14:22:33.282620 6983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:33.283188 6976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:33.283506 6976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:33.284665 6976 hybrid_clock.cc:648] HybridClock initialized: now 1786544553284640 us; error 38 us; skew 500 ppm
Aug 12 14:22:33 dist-test-slave-8w9v krb5kdc[6961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544553, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:33.287456 6976 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:22:33.287958 6976 webserver.cc:460] Webserver started at http://127.5.148.62:40421/ using document root <none> and password file <none>
I20260812 14:22:33.288281 6976 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:33.288355 6976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:33.288519 6976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:33.289785 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "a0dad90cf1744ad797018df1a9243394"
format_stamp: "Formatted at 2026-08-12 14:22:33 on dist-test-slave-8w9v"
server_key: "ad8528e776cef6406034936b963256be"
server_key_iv: "a100a62fbe6e8677d59c8a65375fa08c"
server_key_version: "encryptionkey@0"
I20260812 14:22:33.290185 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "a0dad90cf1744ad797018df1a9243394"
format_stamp: "Formatted at 2026-08-12 14:22:33 on dist-test-slave-8w9v"
server_key: "ad8528e776cef6406034936b963256be"
server_key_iv: "a100a62fbe6e8677d59c8a65375fa08c"
server_key_version: "encryptionkey@0"
I20260812 14:22:33.291769 6976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:22:33.292779 6993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:33.293007 6976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:22:33.293120 6976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "a0dad90cf1744ad797018df1a9243394"
format_stamp: "Formatted at 2026-08-12 14:22:33 on dist-test-slave-8w9v"
server_key: "ad8528e776cef6406034936b963256be"
server_key_iv: "a100a62fbe6e8677d59c8a65375fa08c"
server_key_version: "encryptionkey@0"
I20260812 14:22:33.293200 6976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:33.305930 6976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:33.308059 6976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:33.308283 6976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:33.313848 6976 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:38989
I20260812 14:22:33.313884 7068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:38989 every 8 connection(s)
I20260812 14:22:33.314278 6976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:33.314996 7069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:33.317544 7069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394: Bootstrap starting.
I20260812 14:22:33.318174 7069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:33.318503 7069 log.cc:824] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:33.319239 7069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394: No bootstrap required, opened a new log
I20260812 14:22:33.321692 7069 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dad90cf1744ad797018df1a9243394" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38989 } }
I20260812 14:22:33.321837 7069 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:33.321861 7069 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0dad90cf1744ad797018df1a9243394, State: Initialized, Role: FOLLOWER
I20260812 14:22:33.322072 7069 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dad90cf1744ad797018df1a9243394" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38989 } }
I20260812 14:22:33.322151 7069 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:33.322173 7069 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:33.322249 7069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:33.322841 7069 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dad90cf1744ad797018df1a9243394" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38989 } }
I20260812 14:22:33.322937 7069 leader_election.cc:302] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0dad90cf1744ad797018df1a9243394; no voters:
I20260812 14:22:33.323169 7069 leader_election.cc:288] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:33.323225 7072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:33.323264 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 6976
I20260812 14:22:33.323405 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:33.323413 7072 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [term 1 LEADER]: Becoming Leader. State: Replica: a0dad90cf1744ad797018df1a9243394, State: Running, Role: LEADER
I20260812 14:22:33.323474 7069 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:33.323505 7072 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dad90cf1744ad797018df1a9243394" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38989 } }
I20260812 14:22:33.323593 5712 external_mini_cluster.cc:1428] Setting key 87af02cd5ce4dc6a4a1eb941bc187c94
I20260812 14:22:33.323868 7072 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0dad90cf1744ad797018df1a9243394. Latest consensus state: current_term: 1 leader_uuid: "a0dad90cf1744ad797018df1a9243394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dad90cf1744ad797018df1a9243394" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38989 } } }
I20260812 14:22:33.323937 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0dad90cf1744ad797018df1a9243394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dad90cf1744ad797018df1a9243394" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38989 } } }
I20260812 14:22:33.324028 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:33.324259 7072 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:33.324837 7082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:22:33.325472 7082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:22:33.327071 7082 catalog_manager.cc:1347] Generated new cluster ID: a7b0f4aecb9a43c78f788c4c8e08d237
I20260812 14:22:33.327121 7082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:22:33 dist-test-slave-8w9v krb5kdc[6961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:33.336189 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.324308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.5.148.62:38989)
0812 14:22:33.324493 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:33.324503 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:33.324764 (+ 261us) server_negotiation.cc:407] Connection header received
0812 14:22:33.324937 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:33.324945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:33.325005 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:33.325183 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:33.326083 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.327233 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.328350 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.328952 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.329054 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.331970 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:33.331976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:33.331977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:33.331994 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:33.334143 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.334599 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.334603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.334604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.334738 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.334951 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.334952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.334953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.335318 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:33.335519 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:33.335598 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 14:22:33.335886 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20260812 14:22:33.343340 7082 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:33.344142 7082 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:33.346671 7082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394: Generated new TSK 0
I20260812 14:22:33.346875 7082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:22:33.380012 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:38989
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:33.471189 7098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:33.471488 7098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:33.471539 7098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:33.472816 7098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:33.472892 7098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:33.472936 7098 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:22:33.472977 7098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:22:33.474562 7098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.148.62:38989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:33.474861 7098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:33.475126 7098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:33.480408 7107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:33.480410 7103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:33.480396 7105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:33.480618 7098 server_base.cc:1034] running on GCE node
I20260812 14:22:33.480815 7098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:33.481082 7098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:33.482262 7098 hybrid_clock.cc:648] HybridClock initialized: now 1786544553482244 us; error 42 us; skew 500 ppm
Aug 12 14:22:33 dist-test-slave-8w9v krb5kdc[6961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544553, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:33.485036 7098 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:22:33.485514 7098 webserver.cc:460] Webserver started at http://127.5.148.1:35443/ using document root <none> and password file <none>
I20260812 14:22:33.485791 7098 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:33.485872 7098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:33.486022 7098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:33.487110 7098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "ebf84c7c06e141a391bbf8f17a667955"
format_stamp: "Formatted at 2026-08-12 14:22:33 on dist-test-slave-8w9v"
server_key: "d2b871f06c209f20d9786fb226a95130"
server_key_iv: "b6cbed2561c6dd6207e282fdc10b8bad"
server_key_version: "encryptionkey@0"
I20260812 14:22:33.487488 7098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "ebf84c7c06e141a391bbf8f17a667955"
format_stamp: "Formatted at 2026-08-12 14:22:33 on dist-test-slave-8w9v"
server_key: "d2b871f06c209f20d9786fb226a95130"
server_key_iv: "b6cbed2561c6dd6207e282fdc10b8bad"
server_key_version: "encryptionkey@0"
I20260812 14:22:33.488957 7098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:22:33.489715 7114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:33.489924 7098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:22:33.490044 7098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "ebf84c7c06e141a391bbf8f17a667955"
format_stamp: "Formatted at 2026-08-12 14:22:33 on dist-test-slave-8w9v"
server_key: "d2b871f06c209f20d9786fb226a95130"
server_key_iv: "b6cbed2561c6dd6207e282fdc10b8bad"
server_key_version: "encryptionkey@0"
I20260812 14:22:33.490130 7098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:33.521978 7098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:33.524387 7098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:33.524567 7098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:33.525204 7098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:33.525272 7098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:33.525319 7098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:33.525377 7098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:22:33 dist-test-slave-8w9v krb5kdc[6961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544553, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:33.533442 7098 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:36113
I20260812 14:22:33.533540 7251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:36113 every 8 connection(s)
I20260812 14:22:33.533891 7098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:22:33.536551 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.525983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:59717 (local address 127.5.148.62:38989)
0812 14:22:33.526041 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:33.526043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:33.526318 (+ 275us) server_negotiation.cc:407] Connection header received
0812 14:22:33.527034 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:33.527036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:33.527059 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:33.527135 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:33.528076 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.528644 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.529820 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.530137 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.530216 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.533393 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:33.533394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:33.533394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:33.533406 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:33.535003 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.535465 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.535466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.535466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.535532 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.535805 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.535806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.535806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.535931 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:33.536019 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:33.536363 (+ 344us) server_negotiation.cc:299] Negotiation successful
0812 14:22:33.536458 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:22:33.536746 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.525944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38989 (local address 127.5.148.1:59717)
0812 14:22:33.526154 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:33.526193 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:33.526846 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:33.527251 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:33.527256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:33.527639 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:33.527966 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.527974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.528748 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.528749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.529698 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.529701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.530252 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.530253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.530411 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.531067 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:33.531078 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:33.533227 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:33.535112 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:33.535120 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:33.535122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:33.535374 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:33.535600 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:33.535601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:33.535601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:33.535722 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:33.536011 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:33.536018 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:33.536220 (+ 202us) client_negotiation.cc:769] Sending connection context
0812 14:22:33.536341 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 14:22:33.536509 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
I20260812 14:22:33.537281 7252 heartbeater.cc:347] Connected to a master server at 127.5.148.62:38989
I20260812 14:22:33.537397 7252 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:33.537595 7252 heartbeater.cc:511] Master 127.5.148.62:38989 requested a full tablet report, sending...
I20260812 14:22:33.538229 7012 ts_manager.cc:194] Registered new tserver with Master: ebf84c7c06e141a391bbf8f17a667955 (127.5.148.1:36113)
I20260812 14:22:33.538945 7012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:59717
I20260812 14:22:33.540804 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 7098
I20260812 14:22:33.540940 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:22:33.541126 5712 external_mini_cluster.cc:1428] Setting key f8925bda460ab50af35245980c837b1a
I20260812 14:22:33.544935 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.538465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38989 (local address 127.5.148.1:60737)
0812 14:22:33.538596 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:33.538607 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:33.538665 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:33.538807 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:33.538808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:33.538991 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:33.539077 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.539080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.540012 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.540013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.540757 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.540761 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.541273 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.541274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.541354 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.541802 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:33.541806 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:33.542179 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:33.543699 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:33.543701 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:33.543701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:33.543960 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:33.544196 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:33.544197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:33.544197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:33.544335 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:33.544652 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:33.544653 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:33.544699 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 14:22:33.544769 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 14:22:33.544859 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260812 14:22:33.545017 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.538529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:60737 (local address 127.5.148.62:38989)
0812 14:22:33.538594 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:33.538596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:33.538648 (+ 52us) server_negotiation.cc:407] Connection header received
0812 14:22:33.538721 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:33.538722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:33.538732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:33.538793 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:33.539156 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.539829 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.540851 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.541139 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.541226 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.542261 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:33.542262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:33.542262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:33.542268 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:33.543595 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.544033 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.544034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.544035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.544103 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.544404 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.544405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.544405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.544555 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:33.544633 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:33.544813 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 14:22:33.544905 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:22:33.571461 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:38989
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:33.667742 7263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:33.668047 7263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:33.668102 7263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:33.669546 7263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:33.669641 7263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:33.669677 7263 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:22:33.669723 7263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:22:33.671479 7263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.148.62:38989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:33.671784 7263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:33.672057 7263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:33.677564 7270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:33.677564 7269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:33.677824 7273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:33.678010 7263 server_base.cc:1034] running on GCE node
I20260812 14:22:33.678228 7263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:33.678526 7263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:33.679764 7263 hybrid_clock.cc:648] HybridClock initialized: now 1786544553679739 us; error 47 us; skew 500 ppm
Aug 12 14:22:33 dist-test-slave-8w9v krb5kdc[6961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544553, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:33.682775 7263 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:22:33.683305 7263 webserver.cc:460] Webserver started at http://127.5.148.2:40999/ using document root <none> and password file <none>
I20260812 14:22:33.683530 7263 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:33.683575 7263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:33.683678 7263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:33.684901 7263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "c858455095ef465d9f824f57f02263f0"
format_stamp: "Formatted at 2026-08-12 14:22:33 on dist-test-slave-8w9v"
server_key: "36475bf9cf95a0819295d1a078182988"
server_key_iv: "e254ffa2933b19a13d11f183860d7852"
server_key_version: "encryptionkey@0"
I20260812 14:22:33.685258 7263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "c858455095ef465d9f824f57f02263f0"
format_stamp: "Formatted at 2026-08-12 14:22:33 on dist-test-slave-8w9v"
server_key: "36475bf9cf95a0819295d1a078182988"
server_key_iv: "e254ffa2933b19a13d11f183860d7852"
server_key_version: "encryptionkey@0"
I20260812 14:22:33.686859 7263 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:22:33.687654 7282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:33.687873 7263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:33.688000 7263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "c858455095ef465d9f824f57f02263f0"
format_stamp: "Formatted at 2026-08-12 14:22:33 on dist-test-slave-8w9v"
server_key: "36475bf9cf95a0819295d1a078182988"
server_key_iv: "e254ffa2933b19a13d11f183860d7852"
server_key_version: "encryptionkey@0"
I20260812 14:22:33.688067 7263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:33.717317 7263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:33.719669 7263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:33.719796 7263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:33.720472 7263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:33.720522 7263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:33.720556 7263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:33.720597 7263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:22:33 dist-test-slave-8w9v krb5kdc[6961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544553, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:33.729738 7263 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:34187
I20260812 14:22:33.729882 7416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:34187 every 8 connection(s)
I20260812 14:22:33.730247 7263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:22:33.732280 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.720990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:35021 (local address 127.5.148.62:38989)
0812 14:22:33.721065 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:33.721068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:33.721264 (+ 196us) server_negotiation.cc:407] Connection header received
0812 14:22:33.721817 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:33.721820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:33.721834 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:33.721932 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:33.722604 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.723489 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.725294 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.725669 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.725749 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.729073 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:33.729075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:33.729076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:33.729084 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:33.730889 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.731370 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.731372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.731372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.731431 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.731659 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.731660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.731660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.731771 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:33.731822 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:33.732080 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 14:22:33.732207 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 14:22:33.732467 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.720946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38989 (local address 127.5.148.2:35021)
0812 14:22:33.721116 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:33.721161 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:33.721698 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:33.722001 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:33.722004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:33.722269 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:33.722499 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.722504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.723613 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.723614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.725185 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.725189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.725833 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.725835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.726053 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.726806 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:33.726820 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:33.728958 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:33.730967 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:33.730972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:33.730974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:33.731293 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:33.731495 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:33.731496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:33.731496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:33.731602 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:33.731826 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:33.731830 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:33.731960 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 14:22:33.732045 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 14:22:33.732236 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260812 14:22:33.733027 7417 heartbeater.cc:347] Connected to a master server at 127.5.148.62:38989
I20260812 14:22:33.733119 7417 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:33.733328 7417 heartbeater.cc:511] Master 127.5.148.62:38989 requested a full tablet report, sending...
I20260812 14:22:33.733695 7012 ts_manager.cc:194] Registered new tserver with Master: c858455095ef465d9f824f57f02263f0 (127.5.148.2:34187)
I20260812 14:22:33.734082 7012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:35021
I20260812 14:22:33.738231 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 7263
I20260812 14:22:33.738415 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:22:33.738655 5712 external_mini_cluster.cc:1428] Setting key 1c6d71d3e5bf8aabb8bffb8a523203a2
I20260812 14:22:33.740522 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.734180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38989 (local address 127.5.148.2:56863)
0812 14:22:33.734322 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:33.734334 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:33.734388 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:33.734506 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:33.734507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:33.734715 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:33.734767 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.734768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.735448 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.735449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.735970 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.735972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.736432 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.736433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.736508 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.737016 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:33.737021 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:33.737367 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:33.739167 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:33.739168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:33.739168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:33.739452 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:33.739686 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:33.739688 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:33.739688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:33.739759 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:33.740208 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:33.740209 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:33.740265 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 14:22:33.740331 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 14:22:33.740439 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260812 14:22:33.740607 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.734361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:56863 (local address 127.5.148.62:38989)
0812 14:22:33.734406 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:33.734408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:33.734416 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:22:33.734444 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:33.734445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:33.734455 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:33.734504 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:33.734856 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.735338 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.736074 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.736333 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.736422 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.737466 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:33.737467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:33.737468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:33.737476 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:33.739073 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.739508 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.739509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.739509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.739582 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.739839 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.739840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.739840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.740074 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:33.740182 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:33.740380 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 14:22:33.740474 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:22:33.776235 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:38989
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:33.869843 7426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:33.870057 7426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:33.870087 7426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:33.871273 7426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:33.871318 7426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:33.871340 7426 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:22:33.871359 7426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:22:33.872968 7426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.148.62:38989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:33.873221 7426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:33.873466 7426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:33.879002 7436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:33.879002 7434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:33.879004 7426 server_base.cc:1034] running on GCE node
W20260812 14:22:33.879025 7433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:33.879534 7426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:33.879873 7426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:33.881034 7426 hybrid_clock.cc:648] HybridClock initialized: now 1786544553880995 us; error 59 us; skew 500 ppm
Aug 12 14:22:33 dist-test-slave-8w9v krb5kdc[6961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544553, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:33.883912 7426 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:22:33.884522 7426 webserver.cc:460] Webserver started at http://127.5.148.3:45443/ using document root <none> and password file <none>
I20260812 14:22:33.884807 7426 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:33.884891 7426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:33.885061 7426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:33.886299 7426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "aa7a666a4d8e41609c87503e98c3e0df"
format_stamp: "Formatted at 2026-08-12 14:22:33 on dist-test-slave-8w9v"
server_key: "61f1e99b164ca243e866aecde901f38e"
server_key_iv: "a384282d9c1d7017eabb857dcf14ef1f"
server_key_version: "encryptionkey@0"
I20260812 14:22:33.886721 7426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "aa7a666a4d8e41609c87503e98c3e0df"
format_stamp: "Formatted at 2026-08-12 14:22:33 on dist-test-slave-8w9v"
server_key: "61f1e99b164ca243e866aecde901f38e"
server_key_iv: "a384282d9c1d7017eabb857dcf14ef1f"
server_key_version: "encryptionkey@0"
I20260812 14:22:33.888379 7426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260812 14:22:33.889256 7444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:33.889485 7426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:33.889614 7426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "aa7a666a4d8e41609c87503e98c3e0df"
format_stamp: "Formatted at 2026-08-12 14:22:33 on dist-test-slave-8w9v"
server_key: "61f1e99b164ca243e866aecde901f38e"
server_key_iv: "a384282d9c1d7017eabb857dcf14ef1f"
server_key_version: "encryptionkey@0"
I20260812 14:22:33.889703 7426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:33.902650 7426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:33.904834 7426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:33.905035 7426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:33.905653 7426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:33.905726 7426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:33.905786 7426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:33.905819 7426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:22:33 dist-test-slave-8w9v krb5kdc[6961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544553, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:33.914987 7426 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:36013
I20260812 14:22:33.915408 7426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:22:33.915830 7576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:36013 every 8 connection(s)
I20260812 14:22:33.917390 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.906195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:38401 (local address 127.5.148.62:38989)
0812 14:22:33.906240 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:33.906242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:33.906568 (+ 326us) server_negotiation.cc:407] Connection header received
0812 14:22:33.907219 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:33.907221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:33.907234 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:33.907301 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:33.908248 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.908847 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.910066 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.910270 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.910360 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.913937 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:33.913940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:33.913941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:33.913952 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:33.915802 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.916487 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.916488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.916489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.916550 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.916815 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.916815 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.916816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.916951 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:33.917004 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:33.917249 (+ 245us) server_negotiation.cc:299] Negotiation successful
0812 14:22:33.917323 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:22:33.917615 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.906200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38989 (local address 127.5.148.3:38401)
0812 14:22:33.906439 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:33.906478 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:33.907108 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:33.907381 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:33.907385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:33.907791 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:33.908117 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.908125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.908966 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.908967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.909974 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.909977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.910374 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.910375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.910511 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.911172 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:33.911182 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:33.913805 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:33.915943 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:33.915948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:33.915950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:33.916397 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:33.916630 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:33.916631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:33.916631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:33.916747 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:33.916996 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:33.916999 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:33.917126 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 14:22:33.917207 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 14:22:33.917352 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":137,"threads_started":1}
I20260812 14:22:33.918241 7577 heartbeater.cc:347] Connected to a master server at 127.5.148.62:38989
I20260812 14:22:33.918392 7577 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:33.918648 7577 heartbeater.cc:511] Master 127.5.148.62:38989 requested a full tablet report, sending...
I20260812 14:22:33.919093 7013 ts_manager.cc:194] Registered new tserver with Master: aa7a666a4d8e41609c87503e98c3e0df (127.5.148.3:36013)
I20260812 14:22:33.919669 7013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:38401
I20260812 14:22:33.922665 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 7426
I20260812 14:22:33.922781 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:22:33.922976 5712 external_mini_cluster.cc:1428] Setting key 4bdbc3b13c668869c24c84e7c32bd9a4
I20260812 14:22:33.924045 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:22:33.925385 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.919417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:41553 (local address 127.5.148.62:38989)
0812 14:22:33.919459 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:33.919461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:33.919473 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:22:33.919530 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:33.919530 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:33.919540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:33.919602 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:33.919950 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.920520 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.921212 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.921399 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.921458 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.922338 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:33.922339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:33.922340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:33.922347 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:33.924204 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.924628 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.924629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.924629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.924682 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.924879 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.924879 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.924880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.924975 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:33.925022 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:33.925206 (+ 184us) server_negotiation.cc:299] Negotiation successful
0812 14:22:33.925301 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:22:33.925450 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.919267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38989 (local address 127.5.148.3:41553)
0812 14:22:33.919430 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:33.919441 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:33.919487 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:33.919606 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:33.919606 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:33.919814 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:33.919888 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.919891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.920619 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.920619 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.921129 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.921131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.921473 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.921474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.921536 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.921960 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:33.921964 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:33.922258 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:33.924271 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:33.924273 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:33.924273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:33.924556 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:33.924745 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:33.924746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:33.924746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:33.924819 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:33.925036 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:33.925038 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:33.925095 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 14:22:33.925171 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 14:22:33.925299 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
Aug 12 14:22:33 dist-test-slave-8w9v krb5kdc[6961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544553, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 14:22:33 dist-test-slave-8w9v krb5kdc[6961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544553, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:33.950157 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.943081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52862 (local address 127.5.148.62:38989)
0812 14:22:33.943152 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:33.943154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:33.943165 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:22:33.943196 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:33.943197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:33.943207 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:33.943261 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:33.943864 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.944437 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.945166 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.945349 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.945399 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.947756 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:33.947757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:33.947758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:33.947765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:33.949101 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.949435 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.949436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.949436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.949487 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:33.949659 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:33.949659 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:33.949660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:33.949772 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:33.949848 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:33.949973 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 14:22:33.950040 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 14:22:33.952052 7013 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:22:33.953768 7013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:22:33.960830 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.956362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:36113 (local address 127.0.0.1:37588)
0812 14:22:33.956589 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:33.956653 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:33.956723 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:33.957177 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:33.957178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:33.957179 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:33.957241 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.957242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.958186 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.958187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.959479 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.959483 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.960472 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.960473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.960561 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.960576 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 14:22:33.960676 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 14:22:33.960738 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":28,"thread_start_us":76,"threads_started":1}
I20260812 14:22:33.960876 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.956479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37588 (local address 127.5.148.1:36113)
0812 14:22:33.957004 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:33.957008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:33.957018 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:22:33.957057 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:33.957059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:33.957074 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:33.957334 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:33.957402 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.958095 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.959571 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.960364 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.960468 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.960527 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:33.960708 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 14:22:33.960756 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":124,"threads_started":1}
I20260812 14:22:33.960938 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.956890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37732 (local address 127.5.148.2:34187)
0812 14:22:33.957165 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:33.957170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:33.957180 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:22:33.957223 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:33.957225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:33.957240 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:33.957477 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:33.957533 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.958581 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.959814 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.960573 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.960643 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.960711 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:33.960785 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 14:22:33.960828 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":100,"threads_started":1}
I20260812 14:22:33.961328 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.956409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:36013 (local address 127.0.0.1:45648)
0812 14:22:33.956755 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:33.956767 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:33.956823 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:33.957036 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:33.957039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:33.957041 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:33.957118 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.957121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.958591 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.958592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.959698 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.959700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.961086 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.961087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.961158 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.961162 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:22:33.961212 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:22:33.961255 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":60,"thread_start_us":102,"threads_started":1}
I20260812 14:22:33.961396 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.956518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45648 (local address 127.5.148.3:36013)
0812 14:22:33.956796 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:33.956802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:33.956816 (+ 14us) server_negotiation.cc:407] Connection header received
0812 14:22:33.956902 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:33.956904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:33.956921 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:33.957007 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:33.957300 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.958483 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.959788 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.960368 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.960455 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.960520 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:33.961251 (+ 731us) server_negotiation.cc:299] Negotiation successful
0812 14:22:33.961302 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":142,"threads_started":1}
I20260812 14:22:33.961391 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:33.956640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:34187 (local address 127.0.0.1:37732)
0812 14:22:33.956832 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:33.956841 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:33.956880 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:33.957345 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:33.957346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:33.957348 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:33.957391 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.957392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.958641 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.958642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.959713 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:33.959716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:33.960655 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:33.960656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:33.960712 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:33.960715 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:22:33.960774 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:22:33.960815 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260812 14:22:33.961711 7174 tablet_service.cc:1467] Processing CreateTablet for tablet 9478dd2651ed4a14ace3a4b8e01b35f0 (DEFAULT_TABLE table=test-table [id=627698cb98e74f75b9a63769786ac846]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:33.962122 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9478dd2651ed4a14ace3a4b8e01b35f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:33.962092 7502 tablet_service.cc:1467] Processing CreateTablet for tablet 9478dd2651ed4a14ace3a4b8e01b35f0 (DEFAULT_TABLE table=test-table [id=627698cb98e74f75b9a63769786ac846]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:33.962460 7502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9478dd2651ed4a14ace3a4b8e01b35f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:33.962445 7341 tablet_service.cc:1467] Processing CreateTablet for tablet 9478dd2651ed4a14ace3a4b8e01b35f0 (DEFAULT_TABLE table=test-table [id=627698cb98e74f75b9a63769786ac846]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:33.962811 7341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9478dd2651ed4a14ace3a4b8e01b35f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:33.965914 7603 tablet_bootstrap.cc:492] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df: Bootstrap starting.
I20260812 14:22:33.966285 7605 tablet_bootstrap.cc:492] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955: Bootstrap starting.
I20260812 14:22:33.966341 7604 tablet_bootstrap.cc:492] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0: Bootstrap starting.
I20260812 14:22:33.966935 7605 tablet_bootstrap.cc:654] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:33.967115 7603 tablet_bootstrap.cc:654] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:33.967135 7604 tablet_bootstrap.cc:654] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:33.967332 7605 log.cc:824] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:33.967564 7604 log.cc:824] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:33.967564 7603 log.cc:824] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:33.968291 7605 tablet_bootstrap.cc:492] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955: No bootstrap required, opened a new log
I20260812 14:22:33.968370 7605 ts_tablet_manager.cc:1397] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:22:33.968393 7604 tablet_bootstrap.cc:492] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0: No bootstrap required, opened a new log
I20260812 14:22:33.968451 7604 ts_tablet_manager.cc:1397] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:22:33.968631 7603 tablet_bootstrap.cc:492] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df: No bootstrap required, opened a new log
I20260812 14:22:33.968720 7603 ts_tablet_manager.cc:1397] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:22:33.970541 7603 raft_consensus.cc:348] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:33.970724 7603 raft_consensus.cc:374] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:33.970772 7603 raft_consensus.cc:729] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7a666a4d8e41609c87503e98c3e0df, State: Initialized, Role: FOLLOWER
I20260812 14:22:33.970651 7604 raft_consensus.cc:348] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:33.970651 7605 raft_consensus.cc:348] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:33.970816 7605 raft_consensus.cc:374] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:33.970815 7604 raft_consensus.cc:374] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:33.970863 7604 raft_consensus.cc:729] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c858455095ef465d9f824f57f02263f0, State: Initialized, Role: FOLLOWER
I20260812 14:22:33.970863 7605 raft_consensus.cc:729] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf84c7c06e141a391bbf8f17a667955, State: Initialized, Role: FOLLOWER
I20260812 14:22:33.970918 7603 consensus_queue.cc:260] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:33.971004 7604 consensus_queue.cc:260] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:33.971004 7605 consensus_queue.cc:260] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:33.971252 7603 ts_tablet_manager.cc:1428] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:22:33.971341 7604 ts_tablet_manager.cc:1428] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:22:33.971341 7605 ts_tablet_manager.cc:1428] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:22:33.971400 7577 heartbeater.cc:503] Master 127.5.148.62:38989 was elected leader, sending a full tablet report...
I20260812 14:22:33.971469 7252 heartbeater.cc:503] Master 127.5.148.62:38989 was elected leader, sending a full tablet report...
I20260812 14:22:33.971763 7417 heartbeater.cc:503] Master 127.5.148.62:38989 was elected leader, sending a full tablet report...
W20260812 14:22:33.980937 7418 tablet.cc:2369] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:22:34.035113 7253 tablet.cc:2369] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:22:34.166635 7578 tablet.cc:2369] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:22:34.218118 7611 raft_consensus.cc:482] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:22:34.218317 7611 raft_consensus.cc:504] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.218818 7611 leader_election.cc:288] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebf84c7c06e141a391bbf8f17a667955 (127.5.148.1:36113), aa7a666a4d8e41609c87503e98c3e0df (127.5.148.3:36013)
I20260812 14:22:34.221907 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.219240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:35783 (local address 127.5.148.3:36013)
0812 14:22:34.219289 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:34.219291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:34.219299 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:22:34.219397 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:34.219398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:34.219430 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:34.219504 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:34.219671 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.220210 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.221187 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.221545 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.221629 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.221638 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:34.221794 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 14:22:34.221833 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:22:34.221922 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.219118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:34337 (local address 127.5.148.1:36113)
0812 14:22:34.219197 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:34.219200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:34.219212 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:22:34.219307 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:34.219308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:34.219322 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:34.219406 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:34.219593 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.220089 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.221129 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.221519 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.221622 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.221635 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:34.221802 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 14:22:34.221851 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 14:22:34.221921 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.219070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:36113 (local address 127.5.148.2:34337)
0812 14:22:34.219138 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:34.219151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:34.219203 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:34.219414 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:34.219414 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:34.219416 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:34.219505 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.219508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.220235 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.220236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.221024 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.221026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.221624 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.221625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.221706 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.221709 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:22:34.221793 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 14:22:34.221837 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 14:22:34.221928 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.219103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:36013 (local address 127.5.148.2:35783)
0812 14:22:34.219245 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:34.219256 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:34.219316 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:34.219509 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:34.219510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:34.219512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:34.219594 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.219595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.220290 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.220291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.221085 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.221087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.221631 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.221632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.221714 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.221718 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:22:34.221788 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 14:22:34.221830 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260812 14:22:34.222357 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9478dd2651ed4a14ace3a4b8e01b35f0" candidate_uuid: "c858455095ef465d9f824f57f02263f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf84c7c06e141a391bbf8f17a667955" is_pre_election: true
I20260812 14:22:34.222530 7198 raft_consensus.cc:2454] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c858455095ef465d9f824f57f02263f0 in term 0.
I20260812 14:22:34.222680 7528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9478dd2651ed4a14ace3a4b8e01b35f0" candidate_uuid: "c858455095ef465d9f824f57f02263f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7a666a4d8e41609c87503e98c3e0df" is_pre_election: true
I20260812 14:22:34.222810 7285 leader_election.cc:302] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c858455095ef465d9f824f57f02263f0, ebf84c7c06e141a391bbf8f17a667955; no voters:
I20260812 14:22:34.222867 7528 raft_consensus.cc:2454] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c858455095ef465d9f824f57f02263f0 in term 0.
I20260812 14:22:34.222995 7611 raft_consensus.cc:2781] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:22:34.223060 7611 raft_consensus.cc:482] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:22:34.223145 7611 raft_consensus.cc:3037] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:34.224206 7611 raft_consensus.cc:504] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.224386 7611 leader_election.cc:288] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [CANDIDATE]: Term 1 election: Requested vote from peers ebf84c7c06e141a391bbf8f17a667955 (127.5.148.1:36113), aa7a666a4d8e41609c87503e98c3e0df (127.5.148.3:36013)
I20260812 14:22:34.224606 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9478dd2651ed4a14ace3a4b8e01b35f0" candidate_uuid: "c858455095ef465d9f824f57f02263f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf84c7c06e141a391bbf8f17a667955"
I20260812 14:22:34.224714 7198 raft_consensus.cc:3037] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:34.224725 7528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9478dd2651ed4a14ace3a4b8e01b35f0" candidate_uuid: "c858455095ef465d9f824f57f02263f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7a666a4d8e41609c87503e98c3e0df"
I20260812 14:22:34.224833 7528 raft_consensus.cc:3037] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:34.225523 7198 raft_consensus.cc:2454] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c858455095ef465d9f824f57f02263f0 in term 1.
I20260812 14:22:34.225713 7285 leader_election.cc:302] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c858455095ef465d9f824f57f02263f0, ebf84c7c06e141a391bbf8f17a667955; no voters:
I20260812 14:22:34.225804 7528 raft_consensus.cc:2454] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c858455095ef465d9f824f57f02263f0 in term 1.
I20260812 14:22:34.225838 7611 raft_consensus.cc:2781] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:34.226027 7611 raft_consensus.cc:686] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [term 1 LEADER]: Becoming Leader. State: Replica: c858455095ef465d9f824f57f02263f0, State: Running, Role: LEADER
I20260812 14:22:34.226148 7611 consensus_queue.cc:237] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.227172 7010 catalog_manager.cc:5495] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c858455095ef465d9f824f57f02263f0 (127.5.148.2). New cstate: current_term: 1 leader_uuid: "c858455095ef465d9f824f57f02263f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } health_report { overall_health: UNKNOWN } } }
I20260812 14:22:34.251552 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.248806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.5.148.2:34187)
0812 14:22:34.248855 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:34.248857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:34.248890 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:22:34.249044 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:34.249045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:34.249067 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:34.249158 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:22:34.249333 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.249820 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.250638 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.250844 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.250924 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.251169 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:22:34.251274 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:22:34.251369 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:34.251432 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 14:22:34.251456 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:22:34.253527 7528 raft_consensus.cc:1261] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df [term 1 FOLLOWER]: Refusing update from remote peer c858455095ef465d9f824f57f02263f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:34.253530 7198 raft_consensus.cc:1261] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955 [term 1 FOLLOWER]: Refusing update from remote peer c858455095ef465d9f824f57f02263f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:34.253975 7614 consensus_queue.cc:1035] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:34.254109 7611 consensus_queue.cc:1035] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:34.257014 7617 mvcc.cc:205] Tried to move back new op lower bound from 7317686494219378688 to 7317686494111096832. Current Snapshot: MvccSnapshot[applied={T|T < 7317686494219378688}]
I20260812 14:22:34.257321 7621 mvcc.cc:205] Tried to move back new op lower bound from 7317686494219378688 to 7317686494111096832. Current Snapshot: MvccSnapshot[applied={T|T < 7317686494219378688}]
I20260812 14:22:34.257793 7620 mvcc.cc:205] Tried to move back new op lower bound from 7317686494219378688 to 7317686494111096832. Current Snapshot: MvccSnapshot[applied={T|T < 7317686494219378688}]
I20260812 14:22:34.261886 7012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:34.262049 7012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:34.263113 7012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394: Sending DeleteTablet for 3 replicas of tablet 9478dd2651ed4a14ace3a4b8e01b35f0
I20260812 14:22:34.263705 7502 tablet_service.cc:1514] Processing DeleteTablet for tablet 9478dd2651ed4a14ace3a4b8e01b35f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:34 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:45648
I20260812 14:22:34.263727 7174 tablet_service.cc:1514] Processing DeleteTablet for tablet 9478dd2651ed4a14ace3a4b8e01b35f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:34 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:37588
I20260812 14:22:34.263841 7341 tablet_service.cc:1514] Processing DeleteTablet for tablet 9478dd2651ed4a14ace3a4b8e01b35f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:34 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:37732
I20260812 14:22:34.264153 7632 tablet_replica.cc:330] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955: stopping tablet replica
I20260812 14:22:34.264178 7633 tablet_replica.cc:330] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0: stopping tablet replica
I20260812 14:22:34.264264 7632 raft_consensus.cc:2229] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:34.264302 7633 raft_consensus.cc:2229] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:22:34.264384 7632 raft_consensus.cc:2258] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:34.264415 7633 raft_consensus.cc:2258] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:34.264556 7012 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:22:34.264740 7012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:22:34.264837 7630 tablet_replica.cc:330] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df: stopping tablet replica
I20260812 14:22:34.264911 7633 ts_tablet_manager.cc:1905] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:34.264940 7630 raft_consensus.cc:2229] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:34.265051 7630 raft_consensus.cc:2258] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:34.265532 7632 ts_tablet_manager.cc:1905] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:34.266366 7502 tablet_service.cc:1467] Processing CreateTablet for tablet 1af719569eb44128bbfcea82eed6ac49 (DEFAULT_TABLE table=test-table [id=87539b4813b942e68e6fdd470ccd2533]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:34.266485 7502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1af719569eb44128bbfcea82eed6ac49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:34.266947 7633 ts_tablet_manager.cc:1918] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:22:34.267020 7633 log.cc:1196] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/9478dd2651ed4a14ace3a4b8e01b35f0
I20260812 14:22:34.267287 7633 ts_tablet_manager.cc:1939] T 9478dd2651ed4a14ace3a4b8e01b35f0 P c858455095ef465d9f824f57f02263f0: Deleting consensus metadata
I20260812 14:22:34.267616 7630 ts_tablet_manager.cc:1905] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:34.267810 7632 ts_tablet_manager.cc:1918] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:22:34.267902 7632 log.cc:1196] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/wal/wals/9478dd2651ed4a14ace3a4b8e01b35f0
I20260812 14:22:34.267964 7603 tablet_bootstrap.cc:492] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df: Bootstrap starting.
I20260812 14:22:34.268217 7632 ts_tablet_manager.cc:1939] T 9478dd2651ed4a14ace3a4b8e01b35f0 P ebf84c7c06e141a391bbf8f17a667955: Deleting consensus metadata
I20260812 14:22:34.268298 6997 catalog_manager.cc:4841] TS c858455095ef465d9f824f57f02263f0 (127.5.148.2:34187): tablet 9478dd2651ed4a14ace3a4b8e01b35f0 (table test-table [id=627698cb98e74f75b9a63769786ac846]) successfully deleted
I20260812 14:22:34.268374 7174 tablet_service.cc:1467] Processing CreateTablet for tablet 1af719569eb44128bbfcea82eed6ac49 (DEFAULT_TABLE table=test-table [id=87539b4813b942e68e6fdd470ccd2533]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:34.268504 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1af719569eb44128bbfcea82eed6ac49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:34.268512 7603 tablet_bootstrap.cc:654] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:34.268620 7341 tablet_service.cc:1467] Processing CreateTablet for tablet 1af719569eb44128bbfcea82eed6ac49 (DEFAULT_TABLE table=test-table [id=87539b4813b942e68e6fdd470ccd2533]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:34.268718 7341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1af719569eb44128bbfcea82eed6ac49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:34.269234 7603 tablet_bootstrap.cc:492] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df: No bootstrap required, opened a new log
I20260812 14:22:34.269269 7603 ts_tablet_manager.cc:1397] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:34.269395 7603 raft_consensus.cc:348] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.269459 7603 raft_consensus.cc:374] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:34.269472 7603 raft_consensus.cc:729] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7a666a4d8e41609c87503e98c3e0df, State: Initialized, Role: FOLLOWER
I20260812 14:22:34.269539 7603 consensus_queue.cc:260] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.269677 7603 ts_tablet_manager.cc:1428] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:34.269804 7605 tablet_bootstrap.cc:492] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955: Bootstrap starting.
I20260812 14:22:34.270120 7604 tablet_bootstrap.cc:492] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0: Bootstrap starting.
I20260812 14:22:34.270248 7605 tablet_bootstrap.cc:654] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:34.270562 7604 tablet_bootstrap.cc:654] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:34.271413 6997 catalog_manager.cc:4841] TS ebf84c7c06e141a391bbf8f17a667955 (127.5.148.1:36113): tablet 9478dd2651ed4a14ace3a4b8e01b35f0 (table test-table [id=627698cb98e74f75b9a63769786ac846]) successfully deleted
I20260812 14:22:34.272104 7630 ts_tablet_manager.cc:1918] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:22:34.272153 7604 tablet_bootstrap.cc:492] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0: No bootstrap required, opened a new log
I20260812 14:22:34.272207 7604 ts_tablet_manager.cc:1397] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:22:34.272229 7630 log.cc:1196] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/9478dd2651ed4a14ace3a4b8e01b35f0
I20260812 14:22:34.272601 7630 ts_tablet_manager.cc:1939] T 9478dd2651ed4a14ace3a4b8e01b35f0 P aa7a666a4d8e41609c87503e98c3e0df: Deleting consensus metadata
I20260812 14:22:34.272488 7604 raft_consensus.cc:348] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.272657 7604 raft_consensus.cc:374] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:34.272697 7604 raft_consensus.cc:729] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c858455095ef465d9f824f57f02263f0, State: Initialized, Role: FOLLOWER
I20260812 14:22:34.272748 7604 consensus_queue.cc:260] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.272858 7604 ts_tablet_manager.cc:1428] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:34.273119 6995 catalog_manager.cc:4841] TS aa7a666a4d8e41609c87503e98c3e0df (127.5.148.3:36013): tablet 9478dd2651ed4a14ace3a4b8e01b35f0 (table test-table [id=627698cb98e74f75b9a63769786ac846]) successfully deleted
I20260812 14:22:34.273450 7605 tablet_bootstrap.cc:492] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955: No bootstrap required, opened a new log
I20260812 14:22:34.273523 7605 ts_tablet_manager.cc:1397] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260812 14:22:34.273689 7605 raft_consensus.cc:348] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.273770 7605 raft_consensus.cc:374] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:34.273785 7605 raft_consensus.cc:729] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf84c7c06e141a391bbf8f17a667955, State: Initialized, Role: FOLLOWER
I20260812 14:22:34.273847 7605 consensus_queue.cc:260] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.274125 7605 ts_tablet_manager.cc:1428] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:34.427671 7614 raft_consensus.cc:482] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:22:34.427805 7614 raft_consensus.cc:504] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.428073 7614 leader_election.cc:288] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebf84c7c06e141a391bbf8f17a667955 (127.5.148.1:36113), aa7a666a4d8e41609c87503e98c3e0df (127.5.148.3:36013)
I20260812 14:22:34.428458 7528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af719569eb44128bbfcea82eed6ac49" candidate_uuid: "c858455095ef465d9f824f57f02263f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7a666a4d8e41609c87503e98c3e0df" is_pre_election: true
I20260812 14:22:34.428444 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af719569eb44128bbfcea82eed6ac49" candidate_uuid: "c858455095ef465d9f824f57f02263f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf84c7c06e141a391bbf8f17a667955" is_pre_election: true
I20260812 14:22:34.428597 7528 raft_consensus.cc:2454] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c858455095ef465d9f824f57f02263f0 in term 0.
I20260812 14:22:34.428596 7198 raft_consensus.cc:2454] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c858455095ef465d9f824f57f02263f0 in term 0.
I20260812 14:22:34.428784 7284 leader_election.cc:302] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa7a666a4d8e41609c87503e98c3e0df, c858455095ef465d9f824f57f02263f0; no voters:
I20260812 14:22:34.428921 7614 raft_consensus.cc:2781] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:22:34.428977 7614 raft_consensus.cc:482] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:22:34.428994 7614 raft_consensus.cc:3037] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:34.429850 7614 raft_consensus.cc:504] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.430033 7614 leader_election.cc:288] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [CANDIDATE]: Term 1 election: Requested vote from peers ebf84c7c06e141a391bbf8f17a667955 (127.5.148.1:36113), aa7a666a4d8e41609c87503e98c3e0df (127.5.148.3:36013)
I20260812 14:22:34.430238 7528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af719569eb44128bbfcea82eed6ac49" candidate_uuid: "c858455095ef465d9f824f57f02263f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7a666a4d8e41609c87503e98c3e0df"
I20260812 14:22:34.430269 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af719569eb44128bbfcea82eed6ac49" candidate_uuid: "c858455095ef465d9f824f57f02263f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf84c7c06e141a391bbf8f17a667955"
I20260812 14:22:34.430341 7528 raft_consensus.cc:3037] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:34.430346 7198 raft_consensus.cc:3037] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:34.431291 7198 raft_consensus.cc:2454] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c858455095ef465d9f824f57f02263f0 in term 1.
I20260812 14:22:34.431315 7528 raft_consensus.cc:2454] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c858455095ef465d9f824f57f02263f0 in term 1.
I20260812 14:22:34.431555 7285 leader_election.cc:302] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c858455095ef465d9f824f57f02263f0, ebf84c7c06e141a391bbf8f17a667955; no voters:
I20260812 14:22:34.431694 7614 raft_consensus.cc:2781] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:34.431766 7614 raft_consensus.cc:686] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [term 1 LEADER]: Becoming Leader. State: Replica: c858455095ef465d9f824f57f02263f0, State: Running, Role: LEADER
I20260812 14:22:34.431821 7614 consensus_queue.cc:237] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.432545 7010 catalog_manager.cc:5495] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c858455095ef465d9f824f57f02263f0 (127.5.148.2). New cstate: current_term: 1 leader_uuid: "c858455095ef465d9f824f57f02263f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } health_report { overall_health: UNKNOWN } } }
Aug 12 14:22:34 dist-test-slave-8w9v krb5kdc[6961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544553, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:34.481082 7648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.473043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.5.148.62:38989)
0812 14:22:34.473258 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:34.473260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:34.473359 (+ 99us) server_negotiation.cc:407] Connection header received
0812 14:22:34.473432 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:34.473433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:34.473445 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:34.473529 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:34.474060 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.474623 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.475428 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.475641 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.475689 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.478033 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:34.478035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:34.478035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:34.478042 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:34.479522 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:34.479961 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:34.479962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:34.479962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:34.480022 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:34.480290 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:34.480291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:34.480291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:34.480411 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:34.480490 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:34.480882 (+ 392us) server_negotiation.cc:299] Negotiation successful
0812 14:22:34.480967 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":60,"threads_started":1}
I20260812 14:22:34.481204 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.472950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38989 (local address 127.0.0.1:52866)
0812 14:22:34.473235 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:34.473250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:34.473363 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:34.473532 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:34.473533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:34.473909 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:34.473963 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.473966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.474772 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.474773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.475321 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.475322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.475705 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.475707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.475757 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.476344 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:34.476349 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:34.477943 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:34.479624 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:34.479633 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:34.479636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:34.479884 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:34.480089 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:34.480090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:34.480090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:34.480230 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:34.480536 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:34.480545 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:34.480606 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 14:22:34.480867 (+ 261us) client_negotiation.cc:240] Negotiation successful
0812 14:22:34.481054 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":58,"thread_start_us":95,"threads_started":1}
I20260812 14:22:34.483701 7010 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:52866:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 14:22:34.483983 7010 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:22:34.485747 7502 tablet_service.cc:1467] Processing CreateTablet for tablet 573695268e5047ecb1380cc4eb989e9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1085f5eb4a2f434f9fb21cdcc14b04a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:22:34.485808 7174 tablet_service.cc:1467] Processing CreateTablet for tablet 573695268e5047ecb1380cc4eb989e9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1085f5eb4a2f434f9fb21cdcc14b04a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:22:34.485831 7341 tablet_service.cc:1467] Processing CreateTablet for tablet 573695268e5047ecb1380cc4eb989e9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1085f5eb4a2f434f9fb21cdcc14b04a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:22:34.485993 7341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 573695268e5047ecb1380cc4eb989e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:34.485993 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 573695268e5047ecb1380cc4eb989e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:34.486012 7502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 573695268e5047ecb1380cc4eb989e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:34.487438 7603 tablet_bootstrap.cc:492] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df: Bootstrap starting.
I20260812 14:22:34.487658 7604 tablet_bootstrap.cc:492] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0: Bootstrap starting.
I20260812 14:22:34.487658 7605 tablet_bootstrap.cc:492] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955: Bootstrap starting.
I20260812 14:22:34.488080 7604 tablet_bootstrap.cc:654] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:34.488260 7605 tablet_bootstrap.cc:654] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:34.488309 7603 tablet_bootstrap.cc:654] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:34.488896 7605 tablet_bootstrap.cc:492] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955: No bootstrap required, opened a new log
I20260812 14:22:34.488906 7604 tablet_bootstrap.cc:492] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0: No bootstrap required, opened a new log
I20260812 14:22:34.488956 7605 ts_tablet_manager.cc:1397] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:34.488956 7604 ts_tablet_manager.cc:1397] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:34.489056 7603 tablet_bootstrap.cc:492] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df: No bootstrap required, opened a new log
I20260812 14:22:34.489118 7603 ts_tablet_manager.cc:1397] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:22:34.489091 7604 raft_consensus.cc:348] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.489117 7605 raft_consensus.cc:348] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.489172 7604 raft_consensus.cc:374] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:34.489178 7605 raft_consensus.cc:374] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:34.489203 7604 raft_consensus.cc:729] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c858455095ef465d9f824f57f02263f0, State: Initialized, Role: FOLLOWER
I20260812 14:22:34.489204 7605 raft_consensus.cc:729] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf84c7c06e141a391bbf8f17a667955, State: Initialized, Role: FOLLOWER
I20260812 14:22:34.489252 7605 consensus_queue.cc:260] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.489252 7604 consensus_queue.cc:260] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.489302 7603 raft_consensus.cc:348] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.489346 7605 ts_tablet_manager.cc:1428] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:34.489344 7604 ts_tablet_manager.cc:1428] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 14:22:34.489374 7603 raft_consensus.cc:374] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:34.489396 7603 raft_consensus.cc:729] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7a666a4d8e41609c87503e98c3e0df, State: Initialized, Role: FOLLOWER
I20260812 14:22:34.489459 7603 consensus_queue.cc:260] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.489547 7603 ts_tablet_manager.cc:1428] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 14:22:34.489540 7612 tablet_replica.cc:439] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } } }
I20260812 14:22:34.489634 7612 tablet_replica.cc:442] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:22:34.489714 7610 tablet_replica.cc:439] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } } }
I20260812 14:22:34.489738 7614 tablet_replica.cc:439] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } } }
I20260812 14:22:34.489797 7610 tablet_replica.cc:442] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:22:34.489820 7614 tablet_replica.cc:442] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:22:34.829711 7610 raft_consensus.cc:482] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:22:34.829826 7610 raft_consensus.cc:504] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.830220 7610 leader_election.cc:288] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebf84c7c06e141a391bbf8f17a667955 (127.5.148.1:36113), c858455095ef465d9f824f57f02263f0 (127.5.148.2:34187)
I20260812 14:22:34.830513 7612 raft_consensus.cc:482] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:22:34.830614 7612 raft_consensus.cc:504] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.831036 7612 leader_election.cc:288] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c858455095ef465d9f824f57f02263f0 (127.5.148.2:34187), aa7a666a4d8e41609c87503e98c3e0df (127.5.148.3:36013)
I20260812 14:22:34.834702 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.830710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:34187 (local address 127.5.148.3:39637)
0812 14:22:34.830807 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:34.830818 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:34.830874 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:34.831153 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:34.831154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:34.831157 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:34.831220 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.831223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.832237 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.832238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.833607 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.833610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.834466 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.834467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.834521 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.834525 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:22:34.834570 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:22:34.834597 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":111,"thread_start_us":43,"threads_started":1}
I20260812 14:22:34.834702 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.830513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:36113 (local address 127.5.148.3:60627)
0812 14:22:34.830763 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:34.830776 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:34.830857 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:34.831200 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:34.831200 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:34.831202 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:34.831248 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.831249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.832228 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.832230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.833607 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.833610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.834429 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.834431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.834496 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.834501 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:22:34.834535 (+ 34us) client_negotiation.cc:240] Negotiation successful
0812 14:22:34.834585 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":5,"thread_start_us":69,"threads_started":1}
I20260812 14:22:34.834957 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.830810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:39637 (local address 127.5.148.2:34187)
0812 14:22:34.830965 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:34.830967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:34.830977 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:22:34.831012 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:34.831013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:34.831026 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:34.831092 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:34.831307 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.832085 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.833712 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.834332 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.834400 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.834424 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:34.834839 (+ 415us) server_negotiation.cc:299] Negotiation successful
0812 14:22:34.834891 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260812 14:22:34.835261 7364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573695268e5047ecb1380cc4eb989e9d" candidate_uuid: "aa7a666a4d8e41609c87503e98c3e0df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c858455095ef465d9f824f57f02263f0" is_pre_election: true
I20260812 14:22:34.835454 7364 raft_consensus.cc:2454] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa7a666a4d8e41609c87503e98c3e0df in term 0.
I20260812 14:22:34.835490 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.830585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:60627 (local address 127.5.148.1:36113)
0812 14:22:34.831003 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:34.831005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:34.831014 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:22:34.831040 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:34.831041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:34.831053 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:34.831316 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:34.831355 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.832093 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.833712 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.834329 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.835327 (+ 998us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.835342 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:34.835384 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 14:22:34.835413 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":397,"thread_start_us":82,"threads_started":1}
I20260812 14:22:34.835497 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.831123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:34187 (local address 127.5.148.1:47617)
0812 14:22:34.831357 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:34.831369 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:34.831420 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:34.831686 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:34.831687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:34.831689 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:34.831777 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.831779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.832756 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.832757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.834134 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.834138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.834939 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.834940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.835005 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.835010 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:22:34.835043 (+ 33us) client_negotiation.cc:240] Negotiation successful
0812 14:22:34.835393 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":41,"thread_start_us":102,"threads_started":1}
I20260812 14:22:34.835796 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.831236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:36013 (local address 127.5.148.1:52133)
0812 14:22:34.831584 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:34.831595 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:34.831643 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:34.831834 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:34.831836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:34.831837 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:34.831890 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.831891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.832833 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.832834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.834135 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.834138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.835047 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.835048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.835097 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.835099 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:22:34.835149 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:22:34.835536 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":55,"thread_start_us":35,"threads_started":1}
I20260812 14:22:34.835325 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.831440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:52133 (local address 127.5.148.3:36013)
0812 14:22:34.831528 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:34.831531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:34.831696 (+ 165us) server_negotiation.cc:407] Connection header received
0812 14:22:34.831718 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:34.831718 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:34.831728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:34.831791 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:34.831965 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.832752 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.834262 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.834944 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.834981 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.834991 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:34.835176 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 14:22:34.835224 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":46,"threads_started":1}
I20260812 14:22:34.835901 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.831328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:47617 (local address 127.5.148.2:34187)
0812 14:22:34.831529 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:34.831532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:34.831539 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:22:34.831564 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:34.831565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:34.831578 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:34.831858 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:34.831895 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.832654 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.834273 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.834833 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.835161 (+ 328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.835172 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:34.835188 (+ 16us) server_negotiation.cc:299] Negotiation successful
0812 14:22:34.835226 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
I20260812 14:22:34.835664 7447 leader_election.cc:302] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa7a666a4d8e41609c87503e98c3e0df, c858455095ef465d9f824f57f02263f0; no voters:
I20260812 14:22:34.836083 7364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573695268e5047ecb1380cc4eb989e9d" candidate_uuid: "ebf84c7c06e141a391bbf8f17a667955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c858455095ef465d9f824f57f02263f0" is_pre_election: true
I20260812 14:22:34.836143 7610 raft_consensus.cc:2781] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:22:34.836186 7364 raft_consensus.cc:2454] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebf84c7c06e141a391bbf8f17a667955 in term 0.
I20260812 14:22:34.836195 7610 raft_consensus.cc:482] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:22:34.836225 7610 raft_consensus.cc:3037] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:34.836257 7528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573695268e5047ecb1380cc4eb989e9d" candidate_uuid: "ebf84c7c06e141a391bbf8f17a667955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7a666a4d8e41609c87503e98c3e0df" is_pre_election: true
I20260812 14:22:34.836369 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573695268e5047ecb1380cc4eb989e9d" candidate_uuid: "aa7a666a4d8e41609c87503e98c3e0df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf84c7c06e141a391bbf8f17a667955" is_pre_election: true
I20260812 14:22:34.836448 7198 raft_consensus.cc:2454] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa7a666a4d8e41609c87503e98c3e0df in term 0.
I20260812 14:22:34.836779 7118 leader_election.cc:302] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c858455095ef465d9f824f57f02263f0, ebf84c7c06e141a391bbf8f17a667955; no voters:
I20260812 14:22:34.836917 7612 raft_consensus.cc:2781] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:22:34.836977 7612 raft_consensus.cc:482] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:22:34.837028 7612 raft_consensus.cc:3037] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:34.837246 7610 raft_consensus.cc:504] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.837404 7610 leader_election.cc:288] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [CANDIDATE]: Term 1 election: Requested vote from peers ebf84c7c06e141a391bbf8f17a667955 (127.5.148.1:36113), c858455095ef465d9f824f57f02263f0 (127.5.148.2:34187)
I20260812 14:22:34.837419 7528 raft_consensus.cc:2379] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ebf84c7c06e141a391bbf8f17a667955 in current term 1: Already voted for candidate aa7a666a4d8e41609c87503e98c3e0df in this term.
I20260812 14:22:34.837872 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573695268e5047ecb1380cc4eb989e9d" candidate_uuid: "aa7a666a4d8e41609c87503e98c3e0df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf84c7c06e141a391bbf8f17a667955"
I20260812 14:22:34.838047 7612 raft_consensus.cc:504] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.838109 7364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573695268e5047ecb1380cc4eb989e9d" candidate_uuid: "aa7a666a4d8e41609c87503e98c3e0df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c858455095ef465d9f824f57f02263f0"
I20260812 14:22:34.838160 7198 raft_consensus.cc:2379] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aa7a666a4d8e41609c87503e98c3e0df in current term 1: Already voted for candidate ebf84c7c06e141a391bbf8f17a667955 in this term.
I20260812 14:22:34.838184 7612 leader_election.cc:288] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [CANDIDATE]: Term 1 election: Requested vote from peers c858455095ef465d9f824f57f02263f0 (127.5.148.2:34187), aa7a666a4d8e41609c87503e98c3e0df (127.5.148.3:36013)
I20260812 14:22:34.838187 7364 raft_consensus.cc:3037] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:34.838335 7528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573695268e5047ecb1380cc4eb989e9d" candidate_uuid: "ebf84c7c06e141a391bbf8f17a667955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7a666a4d8e41609c87503e98c3e0df"
I20260812 14:22:34.838419 7528 raft_consensus.cc:2379] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ebf84c7c06e141a391bbf8f17a667955 in current term 1: Already voted for candidate aa7a666a4d8e41609c87503e98c3e0df in this term.
I20260812 14:22:34.838508 7363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573695268e5047ecb1380cc4eb989e9d" candidate_uuid: "ebf84c7c06e141a391bbf8f17a667955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c858455095ef465d9f824f57f02263f0"
I20260812 14:22:34.838784 7118 leader_election.cc:302] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebf84c7c06e141a391bbf8f17a667955; no voters: aa7a666a4d8e41609c87503e98c3e0df, c858455095ef465d9f824f57f02263f0
I20260812 14:22:34.838896 7612 raft_consensus.cc:2726] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260812 14:22:34.839093 7364 raft_consensus.cc:2454] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa7a666a4d8e41609c87503e98c3e0df in term 1.
I20260812 14:22:34.839296 7447 leader_election.cc:302] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: aa7a666a4d8e41609c87503e98c3e0df, c858455095ef465d9f824f57f02263f0; no voters: ebf84c7c06e141a391bbf8f17a667955
I20260812 14:22:34.839401 7610 raft_consensus.cc:2781] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:34.839529 7610 raft_consensus.cc:686] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [term 1 LEADER]: Becoming Leader. State: Replica: aa7a666a4d8e41609c87503e98c3e0df, State: Running, Role: LEADER
I20260812 14:22:34.839584 7610 consensus_queue.cc:237] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } }
I20260812 14:22:34.839962 7664 tablet_replica.cc:439] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df: TxnStatusTablet state changed. Reason: New leader aa7a666a4d8e41609c87503e98c3e0df. Latest consensus state: current_term: 1 leader_uuid: "aa7a666a4d8e41609c87503e98c3e0df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } } }
I20260812 14:22:34.840044 7664 tablet_replica.cc:442] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:22:34.840322 7666 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 14:22:34.840440 7666 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 14:22:34.840390 7010 catalog_manager.cc:5495] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df reported cstate change: term changed from 0 to 1, leader changed from <none> to aa7a666a4d8e41609c87503e98c3e0df (127.5.148.3). New cstate: current_term: 1 leader_uuid: "aa7a666a4d8e41609c87503e98c3e0df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } health_report { overall_health: HEALTHY } } }
I20260812 14:22:34.926486 7614 consensus_queue.cc:1035] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:34.928550 7614 consensus_queue.cc:1035] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:34.934695 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.931687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45650 (local address 127.5.148.3:36013)
0812 14:22:34.931739 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:34.931741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:34.931827 (+ 86us) server_negotiation.cc:407] Connection header received
0812 14:22:34.931939 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:34.931939 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:34.931956 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:34.932013 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:22:34.932181 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.932739 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.933860 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.934073 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.934138 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.934378 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:22:34.934476 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:22:34.934533 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:34.934617 (+ 84us) server_negotiation.cc:299] Negotiation successful
0812 14:22:34.934638 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:22:34.934746 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.931604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:36013 (local address 127.0.0.1:45650)
0812 14:22:34.931770 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:34.931784 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:34.931850 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:34.932021 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:34.932022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:34.932023 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:22:34.932077 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.932079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.932816 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.932817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.933753 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.933757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.934158 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.934159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.934228 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.934247 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:22:34.934545 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:22:34.934555 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 14:22:34.934614 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:22:34.934670 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260812 14:22:34.935972 7364 raft_consensus.cc:1261] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0 [term 1 FOLLOWER]: Refusing update from remote peer aa7a666a4d8e41609c87503e98c3e0df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:34.935981 7198 raft_consensus.cc:1261] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955 [term 1 FOLLOWER]: Refusing update from remote peer aa7a666a4d8e41609c87503e98c3e0df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:34.936259 7664 consensus_queue.cc:1035] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:34.936384 7610 consensus_queue.cc:1035] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df [LEADER]: Connected to new peer: Peer: permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:34.937089 7612 tablet_replica.cc:439] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955: TxnStatusTablet state changed. Reason: New leader aa7a666a4d8e41609c87503e98c3e0df. Latest consensus state: current_term: 1 leader_uuid: "aa7a666a4d8e41609c87503e98c3e0df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } } }
I20260812 14:22:34.937193 7612 tablet_replica.cc:442] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:22:34.937327 7611 tablet_replica.cc:439] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0: TxnStatusTablet state changed. Reason: New leader aa7a666a4d8e41609c87503e98c3e0df. Latest consensus state: current_term: 1 leader_uuid: "aa7a666a4d8e41609c87503e98c3e0df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } } }
I20260812 14:22:34.937408 7611 tablet_replica.cc:442] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:22:34.937695 7664 tablet_replica.cc:439] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa7a666a4d8e41609c87503e98c3e0df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } } }
I20260812 14:22:34.937783 7664 tablet_replica.cc:442] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:22:34.938297 7610 tablet_replica.cc:439] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa7a666a4d8e41609c87503e98c3e0df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } } }
I20260812 14:22:34.938400 7610 tablet_replica.cc:442] T 573695268e5047ecb1380cc4eb989e9d P aa7a666a4d8e41609c87503e98c3e0df: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:22:34.939021 7612 tablet_replica.cc:439] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa7a666a4d8e41609c87503e98c3e0df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } } }
I20260812 14:22:34.939139 7612 tablet_replica.cc:442] T 573695268e5047ecb1380cc4eb989e9d P ebf84c7c06e141a391bbf8f17a667955: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:22:34.940032 7611 tablet_replica.cc:439] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa7a666a4d8e41609c87503e98c3e0df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf84c7c06e141a391bbf8f17a667955" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 36113 } } peers { permanent_uuid: "c858455095ef465d9f824f57f02263f0" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34187 } } peers { permanent_uuid: "aa7a666a4d8e41609c87503e98c3e0df" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 36013 } } }
I20260812 14:22:34.940171 7611 tablet_replica.cc:442] T 573695268e5047ecb1380cc4eb989e9d P c858455095ef465d9f824f57f02263f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:22:34.945120 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.942418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:39111 (local address 127.5.148.3:36013)
0812 14:22:34.942480 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:34.942482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:34.942649 (+ 167us) server_negotiation.cc:407] Connection header received
0812 14:22:34.942718 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:34.942719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:34.942730 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:34.942786 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:22:34.942900 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.943421 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.944386 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.944598 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.944676 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.944861 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:22:34.944895 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:22:34.944957 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:34.945024 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 14:22:34.945047 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:22:34.945142 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.942379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:36013 (local address 127.5.148.2:39111)
0812 14:22:34.942589 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:34.942600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:34.942666 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:34.942788 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:34.942789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:34.942791 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:22:34.942843 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.942845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.943514 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.943515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.944283 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.944287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.944676 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.944678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.944735 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.944754 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:22:34.944970 (+ 216us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:22:34.944972 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:22:34.945011 (+ 39us) client_negotiation.cc:240] Negotiation successful
0812 14:22:34.945065 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":149,"threads_started":1}
I20260812 14:22:34.956643 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.953130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:39879 (local address 127.5.148.2:34187)
0812 14:22:34.953174 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:34.953176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:34.953443 (+ 267us) server_negotiation.cc:407] Connection header received
0812 14:22:34.953542 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:34.953543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:34.953556 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:34.953587 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:22:34.953828 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.954561 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.955823 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.956028 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.956083 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.956407 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:22:34.956436 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:22:34.956479 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:34.956559 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 14:22:34.956580 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:22:34.957293 7687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.953091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:34187 (local address 127.5.148.3:39879)
0812 14:22:34.953381 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:34.953393 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:34.953455 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:34.953657 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:34.953658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:34.953659 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:22:34.953743 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.953745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.954669 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.954670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.955725 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:34.955729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.956142 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.956143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:34.956282 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.956300 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:22:34.956502 (+ 202us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:22:34.956504 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:22:34.956556 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 14:22:34.956608 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":136,"threads_started":1}
I20260812 14:22:34.965634 7012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:34.965763 7012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:34.966686 7012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0dad90cf1744ad797018df1a9243394: Sending DeleteTablet for 3 replicas of tablet 1af719569eb44128bbfcea82eed6ac49
I20260812 14:22:34.966984 7502 tablet_service.cc:1514] Processing DeleteTablet for tablet 1af719569eb44128bbfcea82eed6ac49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:34 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:45648
I20260812 14:22:34.967053 7174 tablet_service.cc:1514] Processing DeleteTablet for tablet 1af719569eb44128bbfcea82eed6ac49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:34 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:37588
I20260812 14:22:34.967332 7690 tablet_replica.cc:330] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955: stopping tablet replica
I20260812 14:22:34.967322 7341 tablet_service.cc:1514] Processing DeleteTablet for tablet 1af719569eb44128bbfcea82eed6ac49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:34 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:37732
I20260812 14:22:34.967396 7690 raft_consensus.cc:2229] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:34.967435 7690 raft_consensus.cc:2258] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:34.967608 7692 tablet_replica.cc:330] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0: stopping tablet replica
I20260812 14:22:34.967674 7692 raft_consensus.cc:2229] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:22:34.967738 7692 raft_consensus.cc:2258] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:34.968106 7692 ts_tablet_manager.cc:1905] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:34.968107 7690 ts_tablet_manager.cc:1905] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:34.968238 7689 tablet_replica.cc:330] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df: stopping tablet replica
I20260812 14:22:34.968312 7689 raft_consensus.cc:2229] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:34.968361 7689 raft_consensus.cc:2258] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:34.968605 7689 ts_tablet_manager.cc:1905] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:34.970335 7689 ts_tablet_manager.cc:1918] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:22:34.970331 7692 ts_tablet_manager.cc:1918] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:22:34.970402 7689 log.cc:1196] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/1af719569eb44128bbfcea82eed6ac49
I20260812 14:22:34.970425 7692 log.cc:1196] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/1af719569eb44128bbfcea82eed6ac49
I20260812 14:22:34.970649 7689 ts_tablet_manager.cc:1939] T 1af719569eb44128bbfcea82eed6ac49 P aa7a666a4d8e41609c87503e98c3e0df: Deleting consensus metadata
I20260812 14:22:34.970685 7692 ts_tablet_manager.cc:1939] T 1af719569eb44128bbfcea82eed6ac49 P c858455095ef465d9f824f57f02263f0: Deleting consensus metadata
I20260812 14:22:34.970662 7690 ts_tablet_manager.cc:1918] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:22:34.970808 7690 log.cc:1196] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786544547303129-5712-0/minicluster-data/ts-0/wal/wals/1af719569eb44128bbfcea82eed6ac49
I20260812 14:22:34.971081 7690 ts_tablet_manager.cc:1939] T 1af719569eb44128bbfcea82eed6ac49 P ebf84c7c06e141a391bbf8f17a667955: Deleting consensus metadata
I20260812 14:22:34.971136 6995 catalog_manager.cc:4841] TS aa7a666a4d8e41609c87503e98c3e0df (127.5.148.3:36013): tablet 1af719569eb44128bbfcea82eed6ac49 (table test-table [id=87539b4813b942e68e6fdd470ccd2533]) successfully deleted
I20260812 14:22:34.971366 6997 catalog_manager.cc:4841] TS c858455095ef465d9f824f57f02263f0 (127.5.148.2:34187): tablet 1af719569eb44128bbfcea82eed6ac49 (table test-table [id=87539b4813b942e68e6fdd470ccd2533]) successfully deleted
I20260812 14:22:34.971547 6997 catalog_manager.cc:4841] TS ebf84c7c06e141a391bbf8f17a667955 (127.5.148.1:36113): tablet 1af719569eb44128bbfcea82eed6ac49 (table test-table [id=87539b4813b942e68e6fdd470ccd2533]) successfully deleted
Aug 12 14:22:34 dist-test-slave-8w9v krb5kdc[6961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544553, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.148.1@KRBTEST.COM
I20260812 14:22:34.979086 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.967883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37590 (local address 127.5.148.1:36113)
0812 14:22:34.967924 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:34.967926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:34.970370 (+ 2444us) server_negotiation.cc:407] Connection header received
0812 14:22:34.970486 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:34.970487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:34.970499 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:34.970530 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:34.971722 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.972281 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.973233 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.973474 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.973560 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.976014 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:34.976026 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:34.976026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:34.976035 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:34.977808 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:34.978204 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:34.978207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:34.978208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:34.978289 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:34.978506 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:34.978507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:34.978507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:34.978665 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:34.978820 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:34.978890 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 14:22:34.978980 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 14:22:34.979420 7245 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37590
I20260812 14:22:34.986603 7648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:34.980540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.5.148.62:38989)
0812 14:22:34.980597 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:34.980600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:34.980618 (+ 18us) server_negotiation.cc:407] Connection header received
0812 14:22:34.980644 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:34.980645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:34.980655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:34.980697 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:34.981214 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.981728 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.982483 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:34.982705 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:34.982793 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:34.984058 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:34.984059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:34.984060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:34.984068 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:34.985409 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:34.985749 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:34.985750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:34.985750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:34.985817 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:34.986023 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:34.986023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:34.986024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:34.986151 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:34.986266 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:34.986402 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 14:22:34.986500 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260812 14:22:34.986891 7012 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52878
I20260812 14:22:34.987421 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 7098
I20260812 14:22:35.001365 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 7263
I20260812 14:22:35.012166 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 7426
I20260812 14:22:35.021703 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 6976
2026-08-12T14:22:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3118 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:22:35 dist-test-slave-8w9v krb5kdc[7699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:22:35 dist-test-slave-8w9v krb5kdc[7699](info): set up 2 sockets
Aug 12 14:22:35 dist-test-slave-8w9v krb5kdc[7699](info): commencing operation
krb5kdc: starting...
W20260812 14:22:36.161381 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:22:36 dist-test-slave-8w9v krb5kdc[7699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:22:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:22:36.291108 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:34153
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:34153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:36.395285 7713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:36.395577 7713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:36.395624 7713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:36.397107 7713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:36.397177 7713 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 14:22:36.397204 7713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:36.397244 7713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:36.397280 7713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:36.397313 7713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:22:36.399004 7713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:34153
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:34153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:36.399336 7713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:36.399597 7713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:36.405047 7718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:36.405035 7719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:36.405015 7721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:36.405352 7713 server_base.cc:1034] running on GCE node
I20260812 14:22:36.405604 7713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:36.405916 7713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:36.407121 7713 hybrid_clock.cc:648] HybridClock initialized: now 1786544556407106 us; error 30 us; skew 500 ppm
Aug 12 14:22:36 dist-test-slave-8w9v krb5kdc[7699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:36.409883 7713 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:22:36.410360 7713 webserver.cc:460] Webserver started at http://127.5.148.62:41579/ using document root <none> and password file <none>
I20260812 14:22:36.410621 7713 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:36.410696 7713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:36.410856 7713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:36.411968 7713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "b623f6c38ae34534af26e792b1c9083d"
format_stamp: "Formatted at 2026-08-12 14:22:36 on dist-test-slave-8w9v"
server_key: "62bf907727df65e60ca858d6e74a9b69"
server_key_iv: "695a15c91c5aa8f17afafd803511abde"
server_key_version: "encryptionkey@0"
I20260812 14:22:36.412388 7713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "b623f6c38ae34534af26e792b1c9083d"
format_stamp: "Formatted at 2026-08-12 14:22:36 on dist-test-slave-8w9v"
server_key: "62bf907727df65e60ca858d6e74a9b69"
server_key_iv: "695a15c91c5aa8f17afafd803511abde"
server_key_version: "encryptionkey@0"
I20260812 14:22:36.413796 7713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:36.414594 7731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:36.414888 7713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:36.415014 7713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "b623f6c38ae34534af26e792b1c9083d"
format_stamp: "Formatted at 2026-08-12 14:22:36 on dist-test-slave-8w9v"
server_key: "62bf907727df65e60ca858d6e74a9b69"
server_key_iv: "695a15c91c5aa8f17afafd803511abde"
server_key_version: "encryptionkey@0"
I20260812 14:22:36.415092 7713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:36.423636 7713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:36.426084 7713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:36.426261 7713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:36.431159 7713 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:34153
I20260812 14:22:36.431285 7807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:34153 every 8 connection(s)
I20260812 14:22:36.431602 7713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:36.432374 7808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:36.434743 7808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d: Bootstrap starting.
I20260812 14:22:36.435377 7808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:36.435678 7808 log.cc:824] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:36.436481 7808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d: No bootstrap required, opened a new log
I20260812 14:22:36.437513 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 7713
I20260812 14:22:36.437608 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:36.437781 5712 external_mini_cluster.cc:1428] Setting key 4895ba5d0df54fcc268272fccd60b143
I20260812 14:22:36.439203 7808 raft_consensus.cc:348] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b623f6c38ae34534af26e792b1c9083d" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 34153 } }
I20260812 14:22:36.439380 7808 raft_consensus.cc:374] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:36.439427 7808 raft_consensus.cc:729] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b623f6c38ae34534af26e792b1c9083d, State: Initialized, Role: FOLLOWER
I20260812 14:22:36.439570 7808 consensus_queue.cc:260] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b623f6c38ae34534af26e792b1c9083d" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 34153 } }
I20260812 14:22:36.439666 7808 raft_consensus.cc:388] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:36.439714 7808 raft_consensus.cc:482] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:36.439776 7808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:36.440470 7808 raft_consensus.cc:504] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b623f6c38ae34534af26e792b1c9083d" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 34153 } }
I20260812 14:22:36.440599 7808 leader_election.cc:302] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b623f6c38ae34534af26e792b1c9083d; no voters:
I20260812 14:22:36.440806 7808 leader_election.cc:288] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:36.440837 7813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:36.440949 7813 raft_consensus.cc:686] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [term 1 LEADER]: Becoming Leader. State: Replica: b623f6c38ae34534af26e792b1c9083d, State: Running, Role: LEADER
I20260812 14:22:36.441093 7808 sys_catalog.cc:564] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:36.441040 7813 consensus_queue.cc:237] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b623f6c38ae34534af26e792b1c9083d" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 34153 } }
I20260812 14:22:36.441488 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b623f6c38ae34534af26e792b1c9083d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b623f6c38ae34534af26e792b1c9083d" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 34153 } } }
I20260812 14:22:36.441498 7815 sys_catalog.cc:455] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b623f6c38ae34534af26e792b1c9083d. Latest consensus state: current_term: 1 leader_uuid: "b623f6c38ae34534af26e792b1c9083d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b623f6c38ae34534af26e792b1c9083d" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 34153 } } }
I20260812 14:22:36.441684 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:36.441704 7815 sys_catalog.cc:458] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:36.442103 7823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:22:36.442683 7823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:22:36.444564 7823 catalog_manager.cc:1347] Generated new cluster ID: 6555219b5cb043149d6620456c26bb9f
I20260812 14:22:36.444621 7823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:22:36 dist-test-slave-8w9v krb5kdc[7699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:36.450291 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:36.438373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.5.148.62:34153)
0812 14:22:36.438585 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:36.438595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:36.438627 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:22:36.439173 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:36.439180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:36.439231 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:36.439384 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:36.440205 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.441421 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.442548 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.443147 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.443282 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:36.446227 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:36.446232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:36.446232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:36.446246 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:36.448435 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:36.448876 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:36.448881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:36.448881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:36.449030 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:36.449249 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:36.449250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:36.449250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:36.449606 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:36.449793 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:36.449888 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 14:22:36.450054 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":112,"threads_started":1}
I20260812 14:22:36.489745 7823 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:36.490355 7823 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:36.494778 7823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d: Generated new TSK 0
I20260812 14:22:36.495020 7823 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:22:36.527279 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:34153
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:36.614877 7837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:36.615099 7837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:36.615125 7837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:36.616380 7837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:36.616425 7837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:36.616453 7837 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 14:22:36.616465 7837 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:22:36.616520 7837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:22:36.618160 7837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.148.62:34153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:36.618422 7837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:36.618675 7837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:36.623962 7849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:36.623968 7845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:36.623945 7846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:36.624352 7837 server_base.cc:1034] running on GCE node
I20260812 14:22:36.624577 7837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:36.624840 7837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:36.626013 7837 hybrid_clock.cc:648] HybridClock initialized: now 1786544556626003 us; error 42 us; skew 500 ppm
Aug 12 14:22:36 dist-test-slave-8w9v krb5kdc[7699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:36.628787 7837 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:22:36.629285 7837 webserver.cc:460] Webserver started at http://127.5.148.1:35951/ using document root <none> and password file <none>
I20260812 14:22:36.629550 7837 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:36.629596 7837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:36.629771 7837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:36.630893 7837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "5a39686ed013474ea71811ea9eaf7cdc"
format_stamp: "Formatted at 2026-08-12 14:22:36 on dist-test-slave-8w9v"
server_key: "95ffc1aabb3b862dcd275082e23362fb"
server_key_iv: "1c3e3d7bf61fdab948442563d0286384"
server_key_version: "encryptionkey@0"
I20260812 14:22:36.631263 7837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "5a39686ed013474ea71811ea9eaf7cdc"
format_stamp: "Formatted at 2026-08-12 14:22:36 on dist-test-slave-8w9v"
server_key: "95ffc1aabb3b862dcd275082e23362fb"
server_key_iv: "1c3e3d7bf61fdab948442563d0286384"
server_key_version: "encryptionkey@0"
I20260812 14:22:36.632769 7837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:22:36.633530 7856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:36.633703 7837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:22:36.633816 7837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "5a39686ed013474ea71811ea9eaf7cdc"
format_stamp: "Formatted at 2026-08-12 14:22:36 on dist-test-slave-8w9v"
server_key: "95ffc1aabb3b862dcd275082e23362fb"
server_key_iv: "1c3e3d7bf61fdab948442563d0286384"
server_key_version: "encryptionkey@0"
I20260812 14:22:36.633903 7837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:36.660629 7837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:36.662866 7837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:36.663061 7837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:36.663686 7837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:36.663766 7837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:36.663810 7837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:36.663857 7837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:22:36 dist-test-slave-8w9v krb5kdc[7699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:36.672673 7837 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:42551
I20260812 14:22:36.672937 7992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:42551 every 8 connection(s)
I20260812 14:22:36.673136 7837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:22:36.673983 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 7837
I20260812 14:22:36.674095 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:22:36.674292 5712 external_mini_cluster.cc:1428] Setting key bfd5eb809111ac07e70d7aa8c81948d1
I20260812 14:22:36.676211 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:36.664448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:52971 (local address 127.5.148.62:34153)
0812 14:22:36.664509 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:36.664512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:36.664771 (+ 259us) server_negotiation.cc:407] Connection header received
0812 14:22:36.665360 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:36.665362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:36.665380 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:36.665434 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:36.666102 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.666929 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.668837 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.669204 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.669288 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:36.672527 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:36.672529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:36.672529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:36.672541 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:36.674540 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:36.675069 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:36.675070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:36.675071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:36.675142 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:36.675421 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:36.675422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:36.675422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:36.675587 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:36.675662 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:36.675921 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 14:22:36.676046 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:22:36.676280 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:36.664421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:34153 (local address 127.5.148.1:52971)
0812 14:22:36.664636 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:36.664679 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:36.665238 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:36.665516 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:36.665520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:36.665784 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:36.665987 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:36.665992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.667069 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.667070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:36.668723 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:36.668727 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.669313 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.669314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:36.669510 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:36.670210 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:36.670222 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:36.672388 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:36.674649 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:36.674657 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:36.674659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:36.674976 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:36.675218 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:36.675219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:36.675219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:36.675325 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:36.675658 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:36.675663 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:36.675788 (+ 125us) client_negotiation.cc:769] Sending connection context
0812 14:22:36.675874 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 14:22:36.676023 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":112,"threads_started":1}
I20260812 14:22:36.677292 7993 heartbeater.cc:347] Connected to a master server at 127.5.148.62:34153
I20260812 14:22:36.677399 7993 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:36.677623 7993 heartbeater.cc:511] Master 127.5.148.62:34153 requested a full tablet report, sending...
I20260812 14:22:36.678201 7749 ts_manager.cc:194] Registered new tserver with Master: 5a39686ed013474ea71811ea9eaf7cdc (127.5.148.1:42551)
I20260812 14:22:36.678879 7749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:52971
I20260812 14:22:36.685441 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:36.678041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:40459 (local address 127.5.148.62:34153)
0812 14:22:36.678081 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:36.678083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:36.678257 (+ 174us) server_negotiation.cc:407] Connection header received
0812 14:22:36.678353 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:36.678354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:36.678365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:36.678396 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:36.679747 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.680322 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.681217 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.681401 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.681477 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:36.682678 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:36.682679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:36.682680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:36.682686 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:36.684243 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:36.684665 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:36.684666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:36.684666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:36.684719 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:36.684944 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:36.684945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:36.684945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:36.685049 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:36.685101 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:36.685292 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 14:22:36.685363 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:22:36.685564 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:36.678042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:34153 (local address 127.5.148.1:40459)
0812 14:22:36.678185 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:36.678200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:36.678283 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:36.678462 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:36.678463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:36.679569 (+ 1106us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:36.679647 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:36.679650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.680431 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.680432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:36.681139 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:36.681141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.681504 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.681506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:36.681591 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:36.682067 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:36.682073 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:36.682556 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:36.684323 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:36.684325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:36.684325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:36.684577 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:36.684783 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:36.684784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:36.684785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:36.684874 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:36.685120 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:36.685122 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:36.685185 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 14:22:36.685267 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 14:22:36.685434 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:22:36.707674 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:34153
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:36.799266 8002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:36.799507 8002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:36.799535 8002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:36.800763 8002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:36.800809 8002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:36.800830 8002 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 14:22:36.800889 8002 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:22:36.800917 8002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:22:36.802440 8002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.148.62:34153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:36.802711 8002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:36.802973 8002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:22:36.808455 8002 server_base.cc:1034] running on GCE node
W20260812 14:22:36.808413 8009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:36.808403 8008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:36.808422 8011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:36.808893 8002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:36.809103 8002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:36.810264 8002 hybrid_clock.cc:648] HybridClock initialized: now 1786544556810252 us; error 28 us; skew 500 ppm
Aug 12 14:22:36 dist-test-slave-8w9v krb5kdc[7699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:36.813009 8002 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:22:36.813547 8002 webserver.cc:460] Webserver started at http://127.5.148.2:43949/ using document root <none> and password file <none>
I20260812 14:22:36.813791 8002 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:36.813838 8002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:36.814013 8002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:36.815147 8002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "70e3ab9792e34f8cbab1136cdf719666"
format_stamp: "Formatted at 2026-08-12 14:22:36 on dist-test-slave-8w9v"
server_key: "a6d813e346e026bfe273bc46e828dce3"
server_key_iv: "c6e5f870e3196c99a1f2b7856ea702b2"
server_key_version: "encryptionkey@0"
I20260812 14:22:36.815527 8002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "70e3ab9792e34f8cbab1136cdf719666"
format_stamp: "Formatted at 2026-08-12 14:22:36 on dist-test-slave-8w9v"
server_key: "a6d813e346e026bfe273bc46e828dce3"
server_key_iv: "c6e5f870e3196c99a1f2b7856ea702b2"
server_key_version: "encryptionkey@0"
I20260812 14:22:36.817061 8002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:22:36.817960 8019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:36.818168 8002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:36.818265 8002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "70e3ab9792e34f8cbab1136cdf719666"
format_stamp: "Formatted at 2026-08-12 14:22:36 on dist-test-slave-8w9v"
server_key: "a6d813e346e026bfe273bc46e828dce3"
server_key_iv: "c6e5f870e3196c99a1f2b7856ea702b2"
server_key_version: "encryptionkey@0"
I20260812 14:22:36.818342 8002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:36.828735 8002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:36.830976 8002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:36.831154 8002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:36.831768 8002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:36.831811 8002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:36.831836 8002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:36.831849 8002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:22:36 dist-test-slave-8w9v krb5kdc[7699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:36.841324 8002 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:41513
I20260812 14:22:36.841542 8150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:41513 every 8 connection(s)
I20260812 14:22:36.841912 8002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:22:36.843883 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:36.832439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:33631 (local address 127.5.148.62:34153)
0812 14:22:36.832515 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:36.832517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:36.832787 (+ 270us) server_negotiation.cc:407] Connection header received
0812 14:22:36.833581 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:36.833582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:36.833593 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:36.833639 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:36.834618 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.835217 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.837179 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.837528 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.837615 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:36.840864 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:36.840865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:36.840866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:36.840873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:36.842750 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:36.843148 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:36.843149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:36.843150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:36.843200 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:36.843374 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:36.843375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:36.843375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:36.843493 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:36.843544 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:36.843749 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 14:22:36.843817 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 14:22:36.844081 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:36.832419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:34153 (local address 127.5.148.2:33631)
0812 14:22:36.832634 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:36.832688 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:36.833460 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:36.833722 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:36.833727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:36.834168 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:36.834489 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:36.834497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.835309 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.835310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:36.837075 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:36.837078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.837661 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.837662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:36.837818 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:36.838486 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:36.838497 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:36.840764 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:36.842826 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:36.842831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:36.842833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:36.843080 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:36.843241 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:36.843241 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:36.843241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:36.843326 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:36.843534 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:36.843537 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:36.843656 (+ 119us) client_negotiation.cc:769] Sending connection context
0812 14:22:36.843736 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 14:22:36.843863 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":110,"threads_started":1}
I20260812 14:22:36.844389 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 8002
I20260812 14:22:36.844503 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:22:36.844712 5712 external_mini_cluster.cc:1428] Setting key 8cf239c96cca0c95c859966cc202f6c9
I20260812 14:22:36.844758 8151 heartbeater.cc:347] Connected to a master server at 127.5.148.62:34153
I20260812 14:22:36.844836 8151 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:36.845041 8151 heartbeater.cc:511] Master 127.5.148.62:34153 requested a full tablet report, sending...
I20260812 14:22:36.845958 7750 ts_manager.cc:194] Registered new tserver with Master: 70e3ab9792e34f8cbab1136cdf719666 (127.5.148.2:41513)
I20260812 14:22:36.846459 7750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:33631
I20260812 14:22:36.853258 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:36.845887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:34153 (local address 127.5.148.2:57241)
0812 14:22:36.846050 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:36.846060 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:36.846112 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:36.846695 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:36.846696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:36.846899 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:36.847061 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:36.847063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.848282 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.848282 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:36.848889 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:36.848891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.849304 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.849305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:36.849438 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:36.849952 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:36.849957 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:36.850258 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:36.852211 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:36.852212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:36.852212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:36.852381 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:36.852593 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:36.852593 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:36.852594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:36.852648 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:36.853007 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:36.853008 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:36.853046 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 14:22:36.853097 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 14:22:36.853179 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
I20260812 14:22:36.853332 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:36.845939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:57241 (local address 127.5.148.62:34153)
0812 14:22:36.846579 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:36.846582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:36.846592 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:22:36.846611 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:36.846612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:36.846623 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:36.846678 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:36.847186 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.848086 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.848976 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:36.849202 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:36.849299 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:36.850372 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:36.850373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:36.850374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:36.850380 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:36.852078 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:36.852457 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:36.852458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:36.852459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:36.852522 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:36.852711 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:36.852713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:36.852713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:36.852928 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:36.853015 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:36.853140 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 14:22:36.853220 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620}
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:22:36.876402 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:34153
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:36.970249 8161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:36.970492 8161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:36.970525 8161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:36.971665 8161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:36.971704 8161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:36.971725 8161 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 14:22:36.971737 8161 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:22:36.971755 8161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:22:36.973382 8161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.148.62:34153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:36.973614 8161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:36.973860 8161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:36.979501 8168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:36.979449 8167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:36.979568 8170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:36.979772 8161 server_base.cc:1034] running on GCE node
I20260812 14:22:36.979992 8161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:36.980317 8161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:36.981489 8161 hybrid_clock.cc:648] HybridClock initialized: now 1786544556981474 us; error 33 us; skew 500 ppm
Aug 12 14:22:36 dist-test-slave-8w9v krb5kdc[7699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:36.984401 8161 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:22:36.984856 8161 webserver.cc:460] Webserver started at http://127.5.148.3:39243/ using document root <none> and password file <none>
I20260812 14:22:36.985071 8161 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:36.985114 8161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:36.985209 8161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:36.986366 8161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "bc914e93afcb41bb941daf7d391da6eb"
format_stamp: "Formatted at 2026-08-12 14:22:36 on dist-test-slave-8w9v"
server_key: "8501dc00d4c4904ae9ced8f8698147fb"
server_key_iv: "961c26dc4fe00376509e342c60947623"
server_key_version: "encryptionkey@0"
I20260812 14:22:36.986699 8161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "bc914e93afcb41bb941daf7d391da6eb"
format_stamp: "Formatted at 2026-08-12 14:22:36 on dist-test-slave-8w9v"
server_key: "8501dc00d4c4904ae9ced8f8698147fb"
server_key_iv: "961c26dc4fe00376509e342c60947623"
server_key_version: "encryptionkey@0"
I20260812 14:22:36.988260 8161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:22:36.989065 8177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:36.989269 8161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:36.989373 8161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "bc914e93afcb41bb941daf7d391da6eb"
format_stamp: "Formatted at 2026-08-12 14:22:36 on dist-test-slave-8w9v"
server_key: "8501dc00d4c4904ae9ced8f8698147fb"
server_key_iv: "961c26dc4fe00376509e342c60947623"
server_key_version: "encryptionkey@0"
I20260812 14:22:36.989463 8161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:36.997532 8161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:37.000041 8161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:37.000244 8161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:37.000752 8161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:37.000787 8161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:37.000815 8161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:37.000872 8161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:22:37 dist-test-slave-8w9v krb5kdc[7699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:37.009724 8161 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:46553
I20260812 14:22:37.009809 8310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:46553 every 8 connection(s)
I20260812 14:22:37.010218 8161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:22:37.012923 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.001348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:50879 (local address 127.5.148.62:34153)
0812 14:22:37.001467 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:37.001470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:37.001780 (+ 310us) server_negotiation.cc:407] Connection header received
0812 14:22:37.002302 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:37.002304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:37.002320 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:37.002422 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:37.003424 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.004302 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.005946 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.006293 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.006379 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.009582 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:37.009584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:37.009584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:37.009594 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:37.011433 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:37.011829 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:37.011830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:37.011831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:37.011884 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:37.012108 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:37.012109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:37.012109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:37.012303 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:37.012363 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:37.012683 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 14:22:37.012807 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 14:22:37.012969 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 8161
I20260812 14:22:37.013005 8187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.001297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:34153 (local address 127.5.148.3:50879)
0812 14:22:37.001650 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:37.001696 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:37.002195 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:37.002494 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:37.002498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:37.002917 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:37.003291 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.003299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.004404 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.004405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.005818 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.005821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.006512 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.006513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.006687 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.007408 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:37.007418 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:37.009428 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:37.011516 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:37.011522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:37.011524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:37.011759 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:37.011944 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:37.011945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:37.011945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:37.012048 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:37.012397 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:37.012404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:37.012549 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 14:22:37.012651 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 14:22:37.012803 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":153,"threads_started":1}
I20260812 14:22:37.013057 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:22:37.013227 5712 external_mini_cluster.cc:1428] Setting key af2bf62afeeeba60c3e4f2d243ab6dd1
I20260812 14:22:37.013756 8311 heartbeater.cc:347] Connected to a master server at 127.5.148.62:34153
I20260812 14:22:37.013890 8311 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:37.014104 8311 heartbeater.cc:511] Master 127.5.148.62:34153 requested a full tablet report, sending...
I20260812 14:22:37.014526 7748 ts_manager.cc:194] Registered new tserver with Master: bc914e93afcb41bb941daf7d391da6eb (127.5.148.3:46553)
I20260812 14:22:37.015071 7748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:50879
I20260812 14:22:37.020921 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.015094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:33727 (local address 127.5.148.62:34153)
0812 14:22:37.015158 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:37.015161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:37.015313 (+ 152us) server_negotiation.cc:407] Connection header received
0812 14:22:37.015387 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:37.015388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:37.015398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:37.015455 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:37.015765 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.016375 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.017063 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.017239 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.017290 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.018195 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:37.018196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:37.018197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:37.018203 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:37.019821 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:37.020175 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:37.020175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:37.020176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:37.020234 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:37.020484 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:37.020485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:37.020486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:37.020624 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:37.020680 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:37.020799 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 14:22:37.020858 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 14:22:37.020948 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.014920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:34153 (local address 127.5.148.3:33727)
0812 14:22:37.015263 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:37.015274 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:37.015330 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:37.015457 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:37.015459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:37.015628 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:37.015685 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.015686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.016463 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.016464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.016985 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.016987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.017307 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.017308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.017372 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.017776 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:37.017782 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:37.018124 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:37.019896 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:37.019897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:37.019897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:37.020063 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:37.020296 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:37.020297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:37.020297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:37.020385 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:37.020687 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:37.020688 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:37.020728 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:22:37.020781 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 14:22:37.020872 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":76,"threads_started":1}
I20260812 14:22:37.025203 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 14:22:37 dist-test-slave-8w9v krb5kdc[7699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 14:22:37 dist-test-slave-8w9v krb5kdc[7699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:37.050521 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.043150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.5.148.62:34153)
0812 14:22:37.043236 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:37.043239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:37.043246 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:22:37.043271 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:37.043272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:37.043282 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:37.043377 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:37.043994 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.044572 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.045488 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.045697 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.045787 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.048277 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:37.048279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:37.048279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:37.048287 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:37.049521 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:37.049855 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:37.049856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:37.049856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:37.049909 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:37.050106 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:37.050106 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:37.050106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:37.050210 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:37.050289 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:37.050385 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 14:22:37.050450 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 14:22:37.052335 7748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:22:37.054003 7748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:22:37.061031 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.056611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:46553 (local address 127.0.0.1:52258)
0812 14:22:37.056972 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:37.056985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:37.057053 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:37.057555 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:37.057556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:37.057558 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:37.057610 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.057612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.058526 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.058527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.059760 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.059763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.060655 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.060656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.060752 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.060776 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 14:22:37.060870 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 14:22:37.060931 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":105,"threads_started":1}
I20260812 14:22:37.061038 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.056709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.5.148.3:46553)
0812 14:22:37.056912 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:37.056917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:37.057342 (+ 425us) server_negotiation.cc:407] Connection header received
0812 14:22:37.057390 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:37.057393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:37.057407 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:37.057482 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:37.057692 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.058428 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.059865 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.060565 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.060656 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.060722 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:37.060892 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 14:22:37.060931 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260812 14:22:37.062269 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.056607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:41513 (local address 127.0.0.1:52058)
0812 14:22:37.056801 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:37.056849 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:37.056923 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:37.057471 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:37.057475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:37.057477 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:37.057562 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.057565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.059003 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.059005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.060104 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.060106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.062001 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.062002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.062072 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.062077 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:22:37.062141 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 14:22:37.062191 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":68,"threads_started":1}
I20260812 14:22:37.062134 8233 tablet_service.cc:1467] Processing CreateTablet for tablet f6d29d1368a6485fb8bb6bbabc2f8cc5 (DEFAULT_TABLE table=test-table [id=e1bac39bf6a847b283bf1447297bc394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:37.062477 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.056822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:42551 (local address 127.0.0.1:33778)
0812 14:22:37.057024 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:37.057052 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:37.057095 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:37.057601 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:37.057602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:37.057604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:37.057641 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.057642 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.060195 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.060200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.060965 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.060967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.062134 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.062135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.062203 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.062208 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:22:37.062240 (+ 32us) client_negotiation.cc:240] Negotiation successful
0812 14:22:37.062283 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
I20260812 14:22:37.062585 8233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d29d1368a6485fb8bb6bbabc2f8cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:37.062584 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.056699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52058 (local address 127.5.148.2:41513)
0812 14:22:37.057226 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:37.057231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:37.057242 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:22:37.057289 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:37.057291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:37.057307 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:37.057399 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:37.057656 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.058888 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.061093 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.061840 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.061975 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.062047 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:37.062414 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 14:22:37.062464 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":89,"threads_started":1}
I20260812 14:22:37.062914 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.057128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.5.148.1:42551)
0812 14:22:37.057329 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:37.057334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:37.057343 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:22:37.057403 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:37.057405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:37.057420 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:37.057718 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:37.057782 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.058887 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.061071 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.061840 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.062636 (+ 796us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.062720 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:37.062753 (+ 33us) server_negotiation.cc:299] Negotiation successful
0812 14:22:37.062800 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":120,"threads_started":1}
I20260812 14:22:37.063225 8074 tablet_service.cc:1467] Processing CreateTablet for tablet f6d29d1368a6485fb8bb6bbabc2f8cc5 (DEFAULT_TABLE table=test-table [id=e1bac39bf6a847b283bf1447297bc394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:37.063508 7916 tablet_service.cc:1467] Processing CreateTablet for tablet f6d29d1368a6485fb8bb6bbabc2f8cc5 (DEFAULT_TABLE table=test-table [id=e1bac39bf6a847b283bf1447297bc394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:37.063643 8074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d29d1368a6485fb8bb6bbabc2f8cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:37.063848 7916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d29d1368a6485fb8bb6bbabc2f8cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:37.066592 8335 tablet_bootstrap.cc:492] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb: Bootstrap starting.
I20260812 14:22:37.067413 8336 tablet_bootstrap.cc:492] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc: Bootstrap starting.
I20260812 14:22:37.067528 8337 tablet_bootstrap.cc:492] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666: Bootstrap starting.
I20260812 14:22:37.068054 8336 tablet_bootstrap.cc:654] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:37.068197 8335 tablet_bootstrap.cc:654] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:37.068202 8337 tablet_bootstrap.cc:654] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:37.068540 8336 log.cc:824] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:37.068621 8337 log.cc:824] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:37.068729 8335 log.cc:824] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:37.069450 8337 tablet_bootstrap.cc:492] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666: No bootstrap required, opened a new log
I20260812 14:22:37.069459 8336 tablet_bootstrap.cc:492] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc: No bootstrap required, opened a new log
I20260812 14:22:37.069526 8336 ts_tablet_manager.cc:1397] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:22:37.069536 8335 tablet_bootstrap.cc:492] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb: No bootstrap required, opened a new log
I20260812 14:22:37.069550 8337 ts_tablet_manager.cc:1397] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:22:37.069593 8335 ts_tablet_manager.cc:1397] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:22:37.071030 8337 raft_consensus.cc:348] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.071199 8337 raft_consensus.cc:374] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:37.071244 8337 raft_consensus.cc:729] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e3ab9792e34f8cbab1136cdf719666, State: Initialized, Role: FOLLOWER
I20260812 14:22:37.071365 8337 consensus_queue.cc:260] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.071703 8337 ts_tablet_manager.cc:1428] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:22:37.071765 8336 raft_consensus.cc:348] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.071820 8335 raft_consensus.cc:348] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.071990 8335 raft_consensus.cc:374] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:37.071985 8336 raft_consensus.cc:374] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:37.072027 8336 raft_consensus.cc:729] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a39686ed013474ea71811ea9eaf7cdc, State: Initialized, Role: FOLLOWER
I20260812 14:22:37.072027 8335 raft_consensus.cc:729] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc914e93afcb41bb941daf7d391da6eb, State: Initialized, Role: FOLLOWER
I20260812 14:22:37.072175 8336 consensus_queue.cc:260] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.072175 8335 consensus_queue.cc:260] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.072216 8151 heartbeater.cc:503] Master 127.5.148.62:34153 was elected leader, sending a full tablet report...
I20260812 14:22:37.072469 8335 ts_tablet_manager.cc:1428] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260812 14:22:37.072469 8336 ts_tablet_manager.cc:1428] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260812 14:22:37.072597 7993 heartbeater.cc:503] Master 127.5.148.62:34153 was elected leader, sending a full tablet report...
I20260812 14:22:37.072628 8311 heartbeater.cc:503] Master 127.5.148.62:34153 was elected leader, sending a full tablet report...
W20260812 14:22:37.092671 8153 tablet.cc:2369] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:22:37.174167 7994 tablet.cc:2369] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:22:37.202090 8342 raft_consensus.cc:482] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:22:37.202296 8342 raft_consensus.cc:504] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.202859 8342 leader_election.cc:288] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a39686ed013474ea71811ea9eaf7cdc (127.5.148.1:42551), bc914e93afcb41bb941daf7d391da6eb (127.5.148.3:46553)
I20260812 14:22:37.206492 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.203230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:37615 (local address 127.5.148.3:46553)
0812 14:22:37.203296 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:37.203298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:37.203305 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:22:37.203331 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:37.203332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:37.203343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:37.203403 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:37.203622 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.204374 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.205536 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.206109 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.206161 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.206170 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:37.206402 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 14:22:37.206437 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 14:22:37.206514 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.203059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:46553 (local address 127.5.148.2:37615)
0812 14:22:37.203107 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:37.203122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:37.203175 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:37.203443 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:37.203444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:37.203446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:37.203531 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.203533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.204462 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.204464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.205440 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.205443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.206195 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.206196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.206312 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.206317 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:22:37.206388 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 14:22:37.206431 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 14:22:37.207015 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.203064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:42551 (local address 127.5.148.2:36815)
0812 14:22:37.203552 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:37.203561 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:37.203640 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:37.203804 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:37.203805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:37.203806 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:37.203971 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.203973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.205013 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.205014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.205948 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.205951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.206709 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.206711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.206786 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.206789 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:22:37.206902 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 14:22:37.206935 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":249,"thread_start_us":161,"threads_started":1}
I20260812 14:22:37.207062 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.203107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:36815 (local address 127.5.148.1:42551)
0812 14:22:37.203156 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:37.203158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:37.203689 (+ 531us) server_negotiation.cc:407] Connection header received
0812 14:22:37.203722 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:37.203723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:37.203735 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:37.203782 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:37.204079 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.204913 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.206051 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.206605 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.206693 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.206705 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:37.206939 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 14:22:37.206985 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:22:37.206997 8257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d29d1368a6485fb8bb6bbabc2f8cc5" candidate_uuid: "70e3ab9792e34f8cbab1136cdf719666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc914e93afcb41bb941daf7d391da6eb" is_pre_election: true
I20260812 14:22:37.207186 8257 raft_consensus.cc:2454] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70e3ab9792e34f8cbab1136cdf719666 in term 0.
I20260812 14:22:37.207471 8023 leader_election.cc:302] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e3ab9792e34f8cbab1136cdf719666, bc914e93afcb41bb941daf7d391da6eb; no voters:
I20260812 14:22:37.207463 7940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d29d1368a6485fb8bb6bbabc2f8cc5" candidate_uuid: "70e3ab9792e34f8cbab1136cdf719666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a39686ed013474ea71811ea9eaf7cdc" is_pre_election: true
I20260812 14:22:37.207617 7940 raft_consensus.cc:2454] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70e3ab9792e34f8cbab1136cdf719666 in term 0.
I20260812 14:22:37.207623 8342 raft_consensus.cc:2781] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:22:37.207671 8342 raft_consensus.cc:482] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:22:37.207710 8342 raft_consensus.cc:3037] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:37.209023 8342 raft_consensus.cc:504] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.209283 8342 leader_election.cc:288] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [CANDIDATE]: Term 1 election: Requested vote from peers 5a39686ed013474ea71811ea9eaf7cdc (127.5.148.1:42551), bc914e93afcb41bb941daf7d391da6eb (127.5.148.3:46553)
I20260812 14:22:37.209401 8257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d29d1368a6485fb8bb6bbabc2f8cc5" candidate_uuid: "70e3ab9792e34f8cbab1136cdf719666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc914e93afcb41bb941daf7d391da6eb"
I20260812 14:22:37.209404 7940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d29d1368a6485fb8bb6bbabc2f8cc5" candidate_uuid: "70e3ab9792e34f8cbab1136cdf719666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a39686ed013474ea71811ea9eaf7cdc"
I20260812 14:22:37.209498 8257 raft_consensus.cc:3037] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:37.209518 7940 raft_consensus.cc:3037] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:37.210357 7940 raft_consensus.cc:2454] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70e3ab9792e34f8cbab1136cdf719666 in term 1.
I20260812 14:22:37.210371 8257 raft_consensus.cc:2454] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70e3ab9792e34f8cbab1136cdf719666 in term 1.
I20260812 14:22:37.210618 8020 leader_election.cc:302] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a39686ed013474ea71811ea9eaf7cdc, 70e3ab9792e34f8cbab1136cdf719666; no voters:
I20260812 14:22:37.210717 8342 raft_consensus.cc:2781] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:37.210872 8342 raft_consensus.cc:686] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [term 1 LEADER]: Becoming Leader. State: Replica: 70e3ab9792e34f8cbab1136cdf719666, State: Running, Role: LEADER
I20260812 14:22:37.210991 8342 consensus_queue.cc:237] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.211969 7749 catalog_manager.cc:5495] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70e3ab9792e34f8cbab1136cdf719666 (127.5.148.2). New cstate: current_term: 1 leader_uuid: "70e3ab9792e34f8cbab1136cdf719666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } health_report { overall_health: UNKNOWN } } }
W20260812 14:22:37.260860 8312 tablet.cc:2369] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 14:22:37 dist-test-slave-8w9v krb5kdc[7699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:37.264276 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.257094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.5.148.62:34153)
0812 14:22:37.257215 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:37.257217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:37.257229 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:22:37.257317 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:37.257318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:37.257328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:37.257383 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:37.257889 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.258387 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.259137 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.259320 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.259377 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.261716 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:37.261717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:37.261718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:37.261724 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:37.263200 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:37.263556 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:37.263557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:37.263557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:37.263610 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:37.263779 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:37.263780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:37.263780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:37.263884 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:37.263951 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:37.264077 (+ 126us) server_negotiation.cc:299] Negotiation successful
0812 14:22:37.264208 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260812 14:22:37.264353 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.256977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:34153 (local address 127.0.0.1:35028)
0812 14:22:37.257148 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:37.257160 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:37.257245 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:37.257412 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:37.257413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:37.257754 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:37.257813 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.257816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.258466 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.258467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.259049 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.259050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.259429 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.259431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.259485 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.260057 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:37.260062 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:37.261624 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:37.263278 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:37.263285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:37.263288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:37.263487 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:37.263670 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:37.263670 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:37.263670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:37.263726 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:37.263952 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:37.263959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:37.264002 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 14:22:37.264059 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 14:22:37.264263 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":109,"thread_start_us":83,"threads_started":1}
I20260812 14:22:37.266837 7749 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:35028:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 14:22:37.267196 7749 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:22:37.269054 8233 tablet_service.cc:1467] Processing CreateTablet for tablet 8842e5546d1d45649e0698e1789e20cf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b08d0563a1154d39b0334820c0310a96]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:22:37.269093 7916 tablet_service.cc:1467] Processing CreateTablet for tablet 8842e5546d1d45649e0698e1789e20cf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b08d0563a1154d39b0334820c0310a96]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:22:37.269160 8074 tablet_service.cc:1467] Processing CreateTablet for tablet 8842e5546d1d45649e0698e1789e20cf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b08d0563a1154d39b0334820c0310a96]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:22:37.269297 8074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8842e5546d1d45649e0698e1789e20cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:37.269297 8233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8842e5546d1d45649e0698e1789e20cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:37.269297 7916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8842e5546d1d45649e0698e1789e20cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:37.270627 8335 tablet_bootstrap.cc:492] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb: Bootstrap starting.
I20260812 14:22:37.270998 8336 tablet_bootstrap.cc:492] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc: Bootstrap starting.
I20260812 14:22:37.271390 8335 tablet_bootstrap.cc:654] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:37.271453 8336 tablet_bootstrap.cc:654] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:37.271880 8337 tablet_bootstrap.cc:492] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666: Bootstrap starting.
I20260812 14:22:37.272338 8336 tablet_bootstrap.cc:492] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc: No bootstrap required, opened a new log
I20260812 14:22:37.272382 8336 ts_tablet_manager.cc:1397] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260812 14:22:37.272477 8335 tablet_bootstrap.cc:492] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb: No bootstrap required, opened a new log
I20260812 14:22:37.272541 8335 ts_tablet_manager.cc:1397] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260812 14:22:37.272562 8336 raft_consensus.cc:348] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.272621 8336 raft_consensus.cc:374] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:37.272635 8336 raft_consensus.cc:729] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a39686ed013474ea71811ea9eaf7cdc, State: Initialized, Role: FOLLOWER
I20260812 14:22:37.272653 8337 tablet_bootstrap.cc:654] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:37.272701 8336 consensus_queue.cc:260] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.272703 8335 raft_consensus.cc:348] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.272790 8335 raft_consensus.cc:374] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:37.272806 8336 ts_tablet_manager.cc:1428] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:37.272811 8335 raft_consensus.cc:729] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc914e93afcb41bb941daf7d391da6eb, State: Initialized, Role: FOLLOWER
I20260812 14:22:37.272869 8335 consensus_queue.cc:260] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.272994 8335 ts_tablet_manager.cc:1428] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:37.273063 8343 tablet_replica.cc:439] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } } }
I20260812 14:22:37.273171 8343 tablet_replica.cc:442] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:22:37.273236 8345 tablet_replica.cc:439] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } } }
I20260812 14:22:37.273336 8345 tablet_replica.cc:442] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:22:37.273707 8337 tablet_bootstrap.cc:492] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666: No bootstrap required, opened a new log
I20260812 14:22:37.273757 8337 ts_tablet_manager.cc:1397] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:22:37.273906 8337 raft_consensus.cc:348] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.274031 8337 raft_consensus.cc:374] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:37.274067 8337 raft_consensus.cc:729] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e3ab9792e34f8cbab1136cdf719666, State: Initialized, Role: FOLLOWER
I20260812 14:22:37.274116 8337 consensus_queue.cc:260] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.274217 8337 ts_tablet_manager.cc:1428] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:37.274281 8342 tablet_replica.cc:439] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } } }
I20260812 14:22:37.274400 8342 tablet_replica.cc:442] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:22:37.282066 8345 raft_consensus.cc:482] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:22:37.282188 8345 raft_consensus.cc:504] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.282645 8345 leader_election.cc:288] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70e3ab9792e34f8cbab1136cdf719666 (127.5.148.2:41513), bc914e93afcb41bb941daf7d391da6eb (127.5.148.3:46553)
I20260812 14:22:37.286958 8360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.282833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:41513 (local address 127.5.148.1:43309)
0812 14:22:37.283042 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:37.283055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:37.283116 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:37.283303 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:37.283304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:37.283306 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:37.283410 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.283412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.284448 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.284449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.286027 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.286032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.286729 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.286730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.286783 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.286787 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:22:37.286838 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 14:22:37.286874 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":57,"thread_start_us":96,"threads_started":1}
I20260812 14:22:37.287002 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.283047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:43309 (local address 127.5.148.2:41513)
0812 14:22:37.283094 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:37.283096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:37.283105 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:22:37.283201 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:37.283202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:37.283216 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:37.283273 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:37.283527 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.284348 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.286117 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.286627 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.286703 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.286714 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:37.286859 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 14:22:37.286907 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:22:37.287288 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.282829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:46553 (local address 127.5.148.1:54007)
0812 14:22:37.283011 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:37.283024 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:37.283128 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:37.283337 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:37.283337 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:37.283338 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:37.283412 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.283413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.284449 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.284450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.286303 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.286306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.287021 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.287023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.287107 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.287113 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 14:22:37.287174 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:22:37.287213 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":60,"thread_start_us":103,"threads_started":1}
I20260812 14:22:37.287287 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.282943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:54007 (local address 127.5.148.3:46553)
0812 14:22:37.283000 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:37.283002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:37.283097 (+ 95us) server_negotiation.cc:407] Connection header received
0812 14:22:37.283219 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:37.283220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:37.283231 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:37.283266 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:37.283527 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.284373 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.286389 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.286933 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.287002 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.287012 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:37.287183 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 14:22:37.287227 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:22:37.287614 8257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8842e5546d1d45649e0698e1789e20cf" candidate_uuid: "5a39686ed013474ea71811ea9eaf7cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc914e93afcb41bb941daf7d391da6eb" is_pre_election: true
I20260812 14:22:37.287744 8257 raft_consensus.cc:2454] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a39686ed013474ea71811ea9eaf7cdc in term 0.
I20260812 14:22:37.287788 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8842e5546d1d45649e0698e1789e20cf" candidate_uuid: "5a39686ed013474ea71811ea9eaf7cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e3ab9792e34f8cbab1136cdf719666" is_pre_election: true
I20260812 14:22:37.287943 8099 raft_consensus.cc:2454] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a39686ed013474ea71811ea9eaf7cdc in term 0.
I20260812 14:22:37.287976 7861 leader_election.cc:302] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a39686ed013474ea71811ea9eaf7cdc, bc914e93afcb41bb941daf7d391da6eb; no voters:
I20260812 14:22:37.288164 8345 raft_consensus.cc:2781] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:22:37.288215 8345 raft_consensus.cc:482] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:22:37.288241 8345 raft_consensus.cc:3037] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:37.289047 8345 raft_consensus.cc:504] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.289228 8345 leader_election.cc:288] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [CANDIDATE]: Term 1 election: Requested vote from peers 70e3ab9792e34f8cbab1136cdf719666 (127.5.148.2:41513), bc914e93afcb41bb941daf7d391da6eb (127.5.148.3:46553)
I20260812 14:22:37.289407 8257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8842e5546d1d45649e0698e1789e20cf" candidate_uuid: "5a39686ed013474ea71811ea9eaf7cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc914e93afcb41bb941daf7d391da6eb"
I20260812 14:22:37.289402 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8842e5546d1d45649e0698e1789e20cf" candidate_uuid: "5a39686ed013474ea71811ea9eaf7cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e3ab9792e34f8cbab1136cdf719666"
I20260812 14:22:37.289500 8257 raft_consensus.cc:3037] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:37.289501 8099 raft_consensus.cc:3037] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:37.290139 8257 raft_consensus.cc:2454] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a39686ed013474ea71811ea9eaf7cdc in term 1.
I20260812 14:22:37.290356 7861 leader_election.cc:302] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a39686ed013474ea71811ea9eaf7cdc, bc914e93afcb41bb941daf7d391da6eb; no voters:
I20260812 14:22:37.290412 8099 raft_consensus.cc:2454] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a39686ed013474ea71811ea9eaf7cdc in term 1.
I20260812 14:22:37.290467 8345 raft_consensus.cc:2781] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:37.290617 8345 raft_consensus.cc:686] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [term 1 LEADER]: Becoming Leader. State: Replica: 5a39686ed013474ea71811ea9eaf7cdc, State: Running, Role: LEADER
I20260812 14:22:37.290716 8345 consensus_queue.cc:237] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.291312 8362 tablet_replica.cc:439] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc: TxnStatusTablet state changed. Reason: New leader 5a39686ed013474ea71811ea9eaf7cdc. Latest consensus state: current_term: 1 leader_uuid: "5a39686ed013474ea71811ea9eaf7cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } } }
I20260812 14:22:37.291419 8362 tablet_replica.cc:442] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:22:37.291649 8364 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 14:22:37.291589 7747 catalog_manager.cc:5495] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a39686ed013474ea71811ea9eaf7cdc (127.5.148.1). New cstate: current_term: 1 leader_uuid: "5a39686ed013474ea71811ea9eaf7cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } health_report { overall_health: UNKNOWN } } }
I20260812 14:22:37.291791 8364 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 14:22:37.297765 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.294829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:42551 (local address 127.0.0.1:33794)
0812 14:22:37.294907 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:37.294921 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:37.294970 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:37.295142 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:37.295143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:37.295144 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:22:37.295191 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.295192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.296105 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.296106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.296757 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.296759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.297247 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.297248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.297299 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.297317 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:22:37.297619 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:22:37.297626 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 14:22:37.297661 (+ 35us) client_negotiation.cc:240] Negotiation successful
0812 14:22:37.297702 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260812 14:22:37.297782 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.294926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33794 (local address 127.5.148.1:42551)
0812 14:22:37.294990 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:37.294992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:37.295004 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:22:37.295039 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:37.295040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:37.295060 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:37.295143 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:22:37.295233 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.295997 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.296849 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.297164 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.297241 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.297421 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:22:37.297552 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:22:37.297639 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:37.297668 (+ 29us) server_negotiation.cc:299] Negotiation successful
0812 14:22:37.297702 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 14:22:37.300215 8099 raft_consensus.cc:1261] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Refusing update from remote peer 5a39686ed013474ea71811ea9eaf7cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:37.300210 8257 raft_consensus.cc:1261] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Refusing update from remote peer 5a39686ed013474ea71811ea9eaf7cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:37.300601 8345 consensus_queue.cc:1035] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:37.300762 8362 consensus_queue.cc:1035] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:37.302129 8342 tablet_replica.cc:439] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666: TxnStatusTablet state changed. Reason: New leader 5a39686ed013474ea71811ea9eaf7cdc. Latest consensus state: current_term: 1 leader_uuid: "5a39686ed013474ea71811ea9eaf7cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } } }
I20260812 14:22:37.302230 8342 tablet_replica.cc:442] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:22:37.302179 8343 tablet_replica.cc:439] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb: TxnStatusTablet state changed. Reason: New leader 5a39686ed013474ea71811ea9eaf7cdc. Latest consensus state: current_term: 1 leader_uuid: "5a39686ed013474ea71811ea9eaf7cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } } }
I20260812 14:22:37.302278 8343 tablet_replica.cc:442] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:22:37.303328 8362 tablet_replica.cc:439] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a39686ed013474ea71811ea9eaf7cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } } }
I20260812 14:22:37.303450 8362 tablet_replica.cc:442] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:22:37.303617 8342 tablet_replica.cc:439] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a39686ed013474ea71811ea9eaf7cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } } }
I20260812 14:22:37.303706 8342 tablet_replica.cc:442] T 8842e5546d1d45649e0698e1789e20cf P 70e3ab9792e34f8cbab1136cdf719666: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:22:37.303862 8371 mvcc.cc:205] Tried to move back new op lower bound from 7317686506697830400 to 7317686506663256064. Current Snapshot: MvccSnapshot[applied={T|T < 7317686506697830400}]
I20260812 14:22:37.303896 8367 mvcc.cc:205] Tried to move back new op lower bound from 7317686506697830400 to 7317686506663256064. Current Snapshot: MvccSnapshot[applied={T|T < 7317686506697830400}]
I20260812 14:22:37.304633 8362 tablet_replica.cc:439] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a39686ed013474ea71811ea9eaf7cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } } }
I20260812 14:22:37.304725 8362 tablet_replica.cc:442] T 8842e5546d1d45649e0698e1789e20cf P 5a39686ed013474ea71811ea9eaf7cdc: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:22:37.305254 8343 tablet_replica.cc:439] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a39686ed013474ea71811ea9eaf7cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } } }
I20260812 14:22:37.305351 8343 tablet_replica.cc:442] T 8842e5546d1d45649e0698e1789e20cf P bc914e93afcb41bb941daf7d391da6eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:22:37.305480 8370 mvcc.cc:205] Tried to move back new op lower bound from 7317686506697830400 to 7317686506663256064. Current Snapshot: MvccSnapshot[applied={T|T < 7317686506697830400}]
I20260812 14:22:37.308887 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.306264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.5.148.2:41513)
0812 14:22:37.306316 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:37.306319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:37.306329 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:22:37.306421 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:37.306422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:37.306442 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:37.306529 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:22:37.306692 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.307184 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.307992 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.308285 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.308362 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.308541 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:22:37.308655 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:22:37.308747 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:37.308773 (+ 26us) server_negotiation.cc:299] Negotiation successful
0812 14:22:37.308797 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:22:37.314237 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.311514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:42551 (local address 127.5.148.2:42831)
0812 14:22:37.311590 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:37.311601 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:37.311644 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:37.311767 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:37.311768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:37.311768 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:22:37.311887 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.311889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.312624 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.312625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.313387 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.313390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.313772 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.313773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.313842 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.313865 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:22:37.314066 (+ 201us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:22:37.314068 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:22:37.314113 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:22:37.314159 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 14:22:37.314237 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.311598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:42831 (local address 127.5.148.1:42551)
0812 14:22:37.311653 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:37.311655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:37.311665 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:22:37.311694 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:37.311694 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:37.311707 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:37.311764 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:22:37.311973 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.312531 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.313472 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.313685 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.313770 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.313968 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:22:37.314006 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:22:37.314060 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:37.314114 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 14:22:37.314139 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:22:37.317220 8257 raft_consensus.cc:1261] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Refusing update from remote peer 70e3ab9792e34f8cbab1136cdf719666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:37.317241 7940 raft_consensus.cc:1261] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Refusing update from remote peer 70e3ab9792e34f8cbab1136cdf719666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:37.317564 8347 consensus_queue.cc:1035] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:37.317754 8342 consensus_queue.cc:1035] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:37.327813 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.324628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:59977 (local address 127.5.148.2:41513)
0812 14:22:37.324699 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:37.324701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:37.324827 (+ 126us) server_negotiation.cc:407] Connection header received
0812 14:22:37.324869 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:37.324870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:37.324883 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:37.324949 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:22:37.325112 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.325859 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.327016 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.327317 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.327393 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.327576 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:22:37.327612 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:22:37.327667 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:37.327713 (+ 46us) server_negotiation.cc:299] Negotiation successful
0812 14:22:37.327734 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 14:22:37.327827 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:37.324588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:41513 (local address 127.5.148.1:59977)
0812 14:22:37.324751 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:37.324763 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:37.324824 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:37.324951 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:37.324951 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:37.324952 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:22:37.325052 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.325054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.325965 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.325967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.326939 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:37.326943 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:37.327396 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:37.327397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:37.327483 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:37.327500 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:22:37.327673 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:22:37.327675 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:22:37.327709 (+ 34us) client_negotiation.cc:240] Negotiation successful
0812 14:22:37.327744 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260812 14:22:37.336776 7749 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:37.336930 7749 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:37.337759 7749 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d: Sending DeleteTablet for 3 replicas of tablet f6d29d1368a6485fb8bb6bbabc2f8cc5
I20260812 14:22:37.338173 8233 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d29d1368a6485fb8bb6bbabc2f8cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:37 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:52258
I20260812 14:22:37.338192 8074 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d29d1368a6485fb8bb6bbabc2f8cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:37 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:52058
I20260812 14:22:37.338495 8389 tablet_replica.cc:330] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb: stopping tablet replica
I20260812 14:22:37.338474 7916 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d29d1368a6485fb8bb6bbabc2f8cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:37 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:33778
I20260812 14:22:37.338617 8389 raft_consensus.cc:2229] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:37.338697 8389 raft_consensus.cc:2258] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:37.338943 8390 tablet_replica.cc:330] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666: stopping tablet replica
I20260812 14:22:37.339062 8390 raft_consensus.cc:2229] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:22:37.339013 7749 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:22:37.339164 7749 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:22:37.339162 8390 raft_consensus.cc:2258] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:37.339696 8389 ts_tablet_manager.cc:1905] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:37.340471 8074 tablet_service.cc:1467] Processing CreateTablet for tablet 6359ab8e08ad49d8b5e721b2465cbdcb (DEFAULT_TABLE table=test-table [id=85a27bfcf8894702a636d9efd67d9e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:37.340502 7916 tablet_service.cc:1467] Processing CreateTablet for tablet 6359ab8e08ad49d8b5e721b2465cbdcb (DEFAULT_TABLE table=test-table [id=85a27bfcf8894702a636d9efd67d9e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:37.340598 8074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6359ab8e08ad49d8b5e721b2465cbdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:37.340601 7916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6359ab8e08ad49d8b5e721b2465cbdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:37.342258 8336 tablet_bootstrap.cc:492] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc: Bootstrap starting.
I20260812 14:22:37.342314 8337 tablet_bootstrap.cc:492] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666: Bootstrap starting.
I20260812 14:22:37.342763 8336 tablet_bootstrap.cc:654] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:37.342810 8337 tablet_bootstrap.cc:654] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:37.343775 8233 tablet_service.cc:1467] Processing CreateTablet for tablet 6359ab8e08ad49d8b5e721b2465cbdcb (DEFAULT_TABLE table=test-table [id=85a27bfcf8894702a636d9efd67d9e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:37.343896 8233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6359ab8e08ad49d8b5e721b2465cbdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:37.345122 8389 ts_tablet_manager.cc:1918] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:22:37.345198 8389 log.cc:1196] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/f6d29d1368a6485fb8bb6bbabc2f8cc5
I20260812 14:22:37.345343 8335 tablet_bootstrap.cc:492] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb: Bootstrap starting.
I20260812 14:22:37.345613 8391 tablet_replica.cc:330] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc: stopping tablet replica
I20260812 14:22:37.345744 8391 raft_consensus.cc:2229] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:37.345772 8389 ts_tablet_manager.cc:1939] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P bc914e93afcb41bb941daf7d391da6eb: Deleting consensus metadata
I20260812 14:22:37.345803 8391 raft_consensus.cc:2258] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:37.345857 8335 tablet_bootstrap.cc:654] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:37.346312 8391 ts_tablet_manager.cc:1905] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:37.346323 8337 tablet_bootstrap.cc:492] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666: No bootstrap required, opened a new log
I20260812 14:22:37.346328 7735 catalog_manager.cc:4841] TS bc914e93afcb41bb941daf7d391da6eb (127.5.148.3:46553): tablet f6d29d1368a6485fb8bb6bbabc2f8cc5 (table test-table [id=e1bac39bf6a847b283bf1447297bc394]) successfully deleted
I20260812 14:22:37.346374 8337 ts_tablet_manager.cc:1397] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260812 14:22:37.346534 8337 raft_consensus.cc:348] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.346617 8337 raft_consensus.cc:374] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:37.346645 8337 raft_consensus.cc:729] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e3ab9792e34f8cbab1136cdf719666, State: Initialized, Role: FOLLOWER
I20260812 14:22:37.346696 8337 consensus_queue.cc:260] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.346724 8390 ts_tablet_manager.cc:1905] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:37.346800 8337 ts_tablet_manager.cc:1428] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 14:22:37.348282 8335 tablet_bootstrap.cc:492] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb: No bootstrap required, opened a new log
I20260812 14:22:37.348384 8335 ts_tablet_manager.cc:1397] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260812 14:22:37.348632 8336 tablet_bootstrap.cc:492] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc: No bootstrap required, opened a new log
I20260812 14:22:37.348687 8336 ts_tablet_manager.cc:1397] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260812 14:22:37.348650 8335 raft_consensus.cc:348] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.348726 8335 raft_consensus.cc:374] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:37.348754 8335 raft_consensus.cc:729] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc914e93afcb41bb941daf7d391da6eb, State: Initialized, Role: FOLLOWER
I20260812 14:22:37.348749 8390 ts_tablet_manager.cc:1918] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:22:37.348825 8390 log.cc:1196] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/f6d29d1368a6485fb8bb6bbabc2f8cc5
I20260812 14:22:37.348803 8335 consensus_queue.cc:260] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.348898 8335 ts_tablet_manager.cc:1428] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 14:22:37.348843 8336 raft_consensus.cc:348] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.349014 8336 raft_consensus.cc:374] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:37.349040 8336 raft_consensus.cc:729] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a39686ed013474ea71811ea9eaf7cdc, State: Initialized, Role: FOLLOWER
I20260812 14:22:37.349103 8390 ts_tablet_manager.cc:1939] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 70e3ab9792e34f8cbab1136cdf719666: Deleting consensus metadata
I20260812 14:22:37.349089 8336 consensus_queue.cc:260] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.349207 8336 ts_tablet_manager.cc:1428] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:37.349542 7732 catalog_manager.cc:4841] TS 70e3ab9792e34f8cbab1136cdf719666 (127.5.148.2:41513): tablet f6d29d1368a6485fb8bb6bbabc2f8cc5 (table test-table [id=e1bac39bf6a847b283bf1447297bc394]) successfully deleted
I20260812 14:22:37.353235 8391 ts_tablet_manager.cc:1918] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:22:37.353345 8391 log.cc:1196] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/wal/wals/f6d29d1368a6485fb8bb6bbabc2f8cc5
I20260812 14:22:37.353629 8391 ts_tablet_manager.cc:1939] T f6d29d1368a6485fb8bb6bbabc2f8cc5 P 5a39686ed013474ea71811ea9eaf7cdc: Deleting consensus metadata
I20260812 14:22:37.354125 7732 catalog_manager.cc:4841] TS 5a39686ed013474ea71811ea9eaf7cdc (127.5.148.1:42551): tablet f6d29d1368a6485fb8bb6bbabc2f8cc5 (table test-table [id=e1bac39bf6a847b283bf1447297bc394]) successfully deleted
I20260812 14:22:37.595587 8384 raft_consensus.cc:482] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:22:37.595717 8384 raft_consensus.cc:504] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.595965 8384 leader_election.cc:288] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a39686ed013474ea71811ea9eaf7cdc (127.5.148.1:42551), bc914e93afcb41bb941daf7d391da6eb (127.5.148.3:46553)
I20260812 14:22:37.596269 7940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6359ab8e08ad49d8b5e721b2465cbdcb" candidate_uuid: "70e3ab9792e34f8cbab1136cdf719666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a39686ed013474ea71811ea9eaf7cdc" is_pre_election: true
I20260812 14:22:37.596312 8257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6359ab8e08ad49d8b5e721b2465cbdcb" candidate_uuid: "70e3ab9792e34f8cbab1136cdf719666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc914e93afcb41bb941daf7d391da6eb" is_pre_election: true
I20260812 14:22:37.596390 7940 raft_consensus.cc:2454] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70e3ab9792e34f8cbab1136cdf719666 in term 0.
I20260812 14:22:37.596398 8257 raft_consensus.cc:2454] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70e3ab9792e34f8cbab1136cdf719666 in term 0.
I20260812 14:22:37.596633 8023 leader_election.cc:302] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e3ab9792e34f8cbab1136cdf719666, bc914e93afcb41bb941daf7d391da6eb; no voters:
I20260812 14:22:37.596750 8384 raft_consensus.cc:2781] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:22:37.596793 8384 raft_consensus.cc:482] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:22:37.596812 8384 raft_consensus.cc:3037] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:37.597602 8384 raft_consensus.cc:504] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.597770 8384 leader_election.cc:288] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [CANDIDATE]: Term 1 election: Requested vote from peers 5a39686ed013474ea71811ea9eaf7cdc (127.5.148.1:42551), bc914e93afcb41bb941daf7d391da6eb (127.5.148.3:46553)
I20260812 14:22:37.597958 7940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6359ab8e08ad49d8b5e721b2465cbdcb" candidate_uuid: "70e3ab9792e34f8cbab1136cdf719666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a39686ed013474ea71811ea9eaf7cdc"
I20260812 14:22:37.597991 8257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6359ab8e08ad49d8b5e721b2465cbdcb" candidate_uuid: "70e3ab9792e34f8cbab1136cdf719666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc914e93afcb41bb941daf7d391da6eb"
I20260812 14:22:37.598084 7940 raft_consensus.cc:3037] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:37.598081 8257 raft_consensus.cc:3037] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:37.598773 8257 raft_consensus.cc:2454] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70e3ab9792e34f8cbab1136cdf719666 in term 1.
I20260812 14:22:37.598783 7940 raft_consensus.cc:2454] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70e3ab9792e34f8cbab1136cdf719666 in term 1.
I20260812 14:22:37.598984 8023 leader_election.cc:302] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e3ab9792e34f8cbab1136cdf719666, bc914e93afcb41bb941daf7d391da6eb; no voters:
I20260812 14:22:37.599104 8384 raft_consensus.cc:2781] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:37.599179 8384 raft_consensus.cc:686] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [term 1 LEADER]: Becoming Leader. State: Replica: 70e3ab9792e34f8cbab1136cdf719666, State: Running, Role: LEADER
I20260812 14:22:37.599227 8384 consensus_queue.cc:237] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:37.599875 7749 catalog_manager.cc:5495] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70e3ab9792e34f8cbab1136cdf719666 (127.5.148.2). New cstate: current_term: 1 leader_uuid: "70e3ab9792e34f8cbab1136cdf719666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } health_report { overall_health: UNKNOWN } } }
I20260812 14:22:37.637655 7940 raft_consensus.cc:1261] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Refusing update from remote peer 70e3ab9792e34f8cbab1136cdf719666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:37.637795 8257 raft_consensus.cc:1261] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Refusing update from remote peer 70e3ab9792e34f8cbab1136cdf719666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:37.638025 8342 consensus_queue.cc:1035] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:37.638151 8384 consensus_queue.cc:1035] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 14:22:37.643514 8371 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 14:22:39.165094 7747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:39.165212 7747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:39.166189 7747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d: Sending DeleteTablet for 3 replicas of tablet 6359ab8e08ad49d8b5e721b2465cbdcb
I20260812 14:22:39.166507 8233 tablet_service.cc:1514] Processing DeleteTablet for tablet 6359ab8e08ad49d8b5e721b2465cbdcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:39 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:52258
I20260812 14:22:39.166523 8074 tablet_service.cc:1514] Processing DeleteTablet for tablet 6359ab8e08ad49d8b5e721b2465cbdcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:39 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:52058
I20260812 14:22:39.168316 7916 tablet_service.cc:1514] Processing DeleteTablet for tablet 6359ab8e08ad49d8b5e721b2465cbdcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:39 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:33778
I20260812 14:22:39.168573 8432 tablet_replica.cc:330] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc: stopping tablet replica
I20260812 14:22:39.168696 8432 raft_consensus.cc:2229] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:39.168823 8432 raft_consensus.cc:2258] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:39.169231 8432 ts_tablet_manager.cc:1905] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:39.171523 8432 ts_tablet_manager.cc:1918] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 14:22:39.171603 8432 log.cc:1196] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/wal/wals/6359ab8e08ad49d8b5e721b2465cbdcb
I20260812 14:22:39.171819 8430 tablet_replica.cc:330] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666: stopping tablet replica
I20260812 14:22:39.171850 8432 ts_tablet_manager.cc:1939] T 6359ab8e08ad49d8b5e721b2465cbdcb P 5a39686ed013474ea71811ea9eaf7cdc: Deleting consensus metadata
I20260812 14:22:39.171902 8430 raft_consensus.cc:2229] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:22:39.172024 8430 raft_consensus.cc:2258] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:39.172188 8429 tablet_replica.cc:330] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb: stopping tablet replica
I20260812 14:22:39.172207 7732 catalog_manager.cc:4841] TS 5a39686ed013474ea71811ea9eaf7cdc (127.5.148.1:42551): tablet 6359ab8e08ad49d8b5e721b2465cbdcb (table test-table [id=85a27bfcf8894702a636d9efd67d9e2d]) successfully deleted
I20260812 14:22:39.172255 8429 raft_consensus.cc:2229] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:39.172302 8429 raft_consensus.cc:2258] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:39.172535 8430 ts_tablet_manager.cc:1905] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:39.172609 8429 ts_tablet_manager.cc:1905] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:39.174121 8429 ts_tablet_manager.cc:1918] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 14:22:39.174209 8429 log.cc:1196] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/6359ab8e08ad49d8b5e721b2465cbdcb
I20260812 14:22:39.174443 8430 ts_tablet_manager.cc:1918] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 14:22:39.174467 8429 ts_tablet_manager.cc:1939] T 6359ab8e08ad49d8b5e721b2465cbdcb P bc914e93afcb41bb941daf7d391da6eb: Deleting consensus metadata
I20260812 14:22:39.174513 8430 log.cc:1196] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/6359ab8e08ad49d8b5e721b2465cbdcb
I20260812 14:22:39.174820 8430 ts_tablet_manager.cc:1939] T 6359ab8e08ad49d8b5e721b2465cbdcb P 70e3ab9792e34f8cbab1136cdf719666: Deleting consensus metadata
I20260812 14:22:39.174855 7735 catalog_manager.cc:4841] TS bc914e93afcb41bb941daf7d391da6eb (127.5.148.3:46553): tablet 6359ab8e08ad49d8b5e721b2465cbdcb (table test-table [id=85a27bfcf8894702a636d9efd67d9e2d]) successfully deleted
I20260812 14:22:39.175232 7732 catalog_manager.cc:4841] TS 70e3ab9792e34f8cbab1136cdf719666 (127.5.148.2:41513): tablet 6359ab8e08ad49d8b5e721b2465cbdcb (table test-table [id=85a27bfcf8894702a636d9efd67d9e2d]) successfully deleted
Aug 12 14:22:39 dist-test-slave-8w9v krb5kdc[7699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544559, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 14:22:39 dist-test-slave-8w9v krb5kdc[7699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544559, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:39.194363 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:39.186083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.5.148.62:34153)
0812 14:22:39.186245 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:39.186248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:39.186259 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:22:39.186339 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:39.186340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:39.186356 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:39.186447 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:39.187054 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.187594 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.188447 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.188662 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.188743 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:39.191322 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:39.191323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:39.191323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:39.191333 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:39.193089 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:39.193481 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:39.193482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:39.193483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:39.193548 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:39.193762 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:39.193763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:39.193764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:39.193891 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:39.194007 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:39.194141 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 14:22:39.194237 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260812 14:22:39.196100 7747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:22:39.196388 7747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:22:39.197953 8074 tablet_service.cc:1467] Processing CreateTablet for tablet 45c65f79fcb847f48fc93d6a6b2ff506 (DEFAULT_TABLE table=test-table [id=19a50f54b07c4970a609f633d058e1e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:39.197933 7916 tablet_service.cc:1467] Processing CreateTablet for tablet 45c65f79fcb847f48fc93d6a6b2ff506 (DEFAULT_TABLE table=test-table [id=19a50f54b07c4970a609f633d058e1e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:39.197969 8233 tablet_service.cc:1467] Processing CreateTablet for tablet 45c65f79fcb847f48fc93d6a6b2ff506 (DEFAULT_TABLE table=test-table [id=19a50f54b07c4970a609f633d058e1e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:39.198098 8074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45c65f79fcb847f48fc93d6a6b2ff506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:39.198097 7916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45c65f79fcb847f48fc93d6a6b2ff506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:39.198132 8233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45c65f79fcb847f48fc93d6a6b2ff506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:39.199455 8440 tablet_bootstrap.cc:492] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb: Bootstrap starting.
I20260812 14:22:39.199564 8442 tablet_bootstrap.cc:492] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666: Bootstrap starting.
I20260812 14:22:39.199919 8440 tablet_bootstrap.cc:654] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:39.199940 8442 tablet_bootstrap.cc:654] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:39.200327 8443 tablet_bootstrap.cc:492] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc: Bootstrap starting.
I20260812 14:22:39.200618 8440 tablet_bootstrap.cc:492] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb: No bootstrap required, opened a new log
I20260812 14:22:39.200685 8440 ts_tablet_manager.cc:1397] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:39.200699 8442 tablet_bootstrap.cc:492] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666: No bootstrap required, opened a new log
I20260812 14:22:39.200713 8443 tablet_bootstrap.cc:654] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:39.200753 8442 ts_tablet_manager.cc:1397] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:39.200863 8442 raft_consensus.cc:348] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.200850 8440 raft_consensus.cc:348] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.200951 8442 raft_consensus.cc:374] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:39.200961 8440 raft_consensus.cc:374] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:39.200975 8442 raft_consensus.cc:729] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e3ab9792e34f8cbab1136cdf719666, State: Initialized, Role: FOLLOWER
I20260812 14:22:39.200990 8440 raft_consensus.cc:729] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc914e93afcb41bb941daf7d391da6eb, State: Initialized, Role: FOLLOWER
I20260812 14:22:39.201038 8440 consensus_queue.cc:260] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.201032 8442 consensus_queue.cc:260] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.201159 8442 ts_tablet_manager.cc:1428] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:39.201187 8440 ts_tablet_manager.cc:1428] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:39.201695 8443 tablet_bootstrap.cc:492] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc: No bootstrap required, opened a new log
I20260812 14:22:39.201783 8443 ts_tablet_manager.cc:1397] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:39.201920 8443 raft_consensus.cc:348] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.202003 8443 raft_consensus.cc:374] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:39.202028 8443 raft_consensus.cc:729] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a39686ed013474ea71811ea9eaf7cdc, State: Initialized, Role: FOLLOWER
I20260812 14:22:39.202081 8443 consensus_queue.cc:260] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.202184 8443 ts_tablet_manager.cc:1428] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:39.434877 8447 raft_consensus.cc:482] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:22:39.434998 8447 raft_consensus.cc:504] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.435324 8447 leader_election.cc:288] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a39686ed013474ea71811ea9eaf7cdc (127.5.148.1:42551), 70e3ab9792e34f8cbab1136cdf719666 (127.5.148.2:41513)
I20260812 14:22:39.439931 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:39.435652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:44101 (local address 127.5.148.1:42551)
0812 14:22:39.435846 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:39.435849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:39.435879 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:22:39.435947 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:39.435949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:39.435961 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:39.436086 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:39.436288 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.437092 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.438903 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.439481 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.439560 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:39.439573 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:39.439806 (+ 233us) server_negotiation.cc:299] Negotiation successful
0812 14:22:39.439850 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":123,"threads_started":1}
I20260812 14:22:39.439931 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:39.435976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:51241 (local address 127.5.148.2:41513)
0812 14:22:39.436169 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:39.436171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:39.436178 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:22:39.436211 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:39.436212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:39.436219 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:39.436307 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:39.436439 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.437172 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.438897 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.439479 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.439560 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:39.439573 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:39.439808 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 14:22:39.439866 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":105,"threads_started":1}
I20260812 14:22:39.439962 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:39.435871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:41513 (local address 127.5.148.3:51241)
0812 14:22:39.436010 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:39.436018 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:39.436071 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:39.436322 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:39.436323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:39.436325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:39.436369 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:39.436370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.437290 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.437291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:39.438767 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:39.438771 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.439593 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.439595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:39.439696 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:39.439704 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 14:22:39.439794 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 14:22:39.439850 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":123,"thread_start_us":77,"threads_started":1}
I20260812 14:22:39.439970 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:39.435536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:42551 (local address 127.5.148.3:44101)
0812 14:22:39.435798 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:39.435810 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:39.435867 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:39.436101 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:39.436102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:39.436105 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:39.436213 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:39.436215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.437203 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.437232 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:39.438767 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:39.438771 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.439597 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.439598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:39.439696 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:39.439705 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 14:22:39.439780 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 14:22:39.439845 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":108,"thread_start_us":198,"threads_started":1}
I20260812 14:22:39.440276 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c65f79fcb847f48fc93d6a6b2ff506" candidate_uuid: "bc914e93afcb41bb941daf7d391da6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e3ab9792e34f8cbab1136cdf719666" is_pre_election: true
I20260812 14:22:39.440389 8099 raft_consensus.cc:2454] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc914e93afcb41bb941daf7d391da6eb in term 0.
I20260812 14:22:39.440563 8178 leader_election.cc:302] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e3ab9792e34f8cbab1136cdf719666, bc914e93afcb41bb941daf7d391da6eb; no voters:
I20260812 14:22:39.440591 7940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c65f79fcb847f48fc93d6a6b2ff506" candidate_uuid: "bc914e93afcb41bb941daf7d391da6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a39686ed013474ea71811ea9eaf7cdc" is_pre_election: true
I20260812 14:22:39.440702 7940 raft_consensus.cc:2454] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc914e93afcb41bb941daf7d391da6eb in term 0.
I20260812 14:22:39.440755 8447 raft_consensus.cc:2781] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:22:39.440806 8447 raft_consensus.cc:482] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:22:39.440843 8447 raft_consensus.cc:3037] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:39.441852 8447 raft_consensus.cc:504] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.442059 8447 leader_election.cc:288] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [CANDIDATE]: Term 1 election: Requested vote from peers 5a39686ed013474ea71811ea9eaf7cdc (127.5.148.1:42551), 70e3ab9792e34f8cbab1136cdf719666 (127.5.148.2:41513)
I20260812 14:22:39.442287 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c65f79fcb847f48fc93d6a6b2ff506" candidate_uuid: "bc914e93afcb41bb941daf7d391da6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e3ab9792e34f8cbab1136cdf719666"
I20260812 14:22:39.442384 8099 raft_consensus.cc:3037] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:39.442322 7940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c65f79fcb847f48fc93d6a6b2ff506" candidate_uuid: "bc914e93afcb41bb941daf7d391da6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a39686ed013474ea71811ea9eaf7cdc"
I20260812 14:22:39.442435 7940 raft_consensus.cc:3037] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:39.443100 8099 raft_consensus.cc:2454] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc914e93afcb41bb941daf7d391da6eb in term 1.
I20260812 14:22:39.443279 8178 leader_election.cc:302] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e3ab9792e34f8cbab1136cdf719666, bc914e93afcb41bb941daf7d391da6eb; no voters:
I20260812 14:22:39.443286 7940 raft_consensus.cc:2454] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc914e93afcb41bb941daf7d391da6eb in term 1.
I20260812 14:22:39.443393 8447 raft_consensus.cc:2781] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:39.443558 8447 raft_consensus.cc:686] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [term 1 LEADER]: Becoming Leader. State: Replica: bc914e93afcb41bb941daf7d391da6eb, State: Running, Role: LEADER
I20260812 14:22:39.443624 8447 consensus_queue.cc:237] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.444380 7747 catalog_manager.cc:5495] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb reported cstate change: term changed from 0 to 1, leader changed from <none> to bc914e93afcb41bb941daf7d391da6eb (127.5.148.3). New cstate: current_term: 1 leader_uuid: "bc914e93afcb41bb941daf7d391da6eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } health_report { overall_health: HEALTHY } } }
I20260812 14:22:39.490375 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:39.487332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.5.148.3:46553)
0812 14:22:39.487543 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:39.487545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:39.487558 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:22:39.487599 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:39.487600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:39.487623 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:39.487725 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:22:39.487891 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.488489 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.489354 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.489614 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.489704 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:39.489920 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:22:39.490065 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:22:39.490163 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:39.490210 (+ 47us) server_negotiation.cc:299] Negotiation successful
0812 14:22:39.490236 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":122,"threads_started":1}
I20260812 14:22:39.495569 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:39.493280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:54653 (local address 127.5.148.1:42551)
0812 14:22:39.493369 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:39.493371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:39.493380 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:22:39.493449 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:39.493450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:39.493462 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:39.493513 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:22:39.493682 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.494160 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.494959 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.495149 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.495203 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:39.495366 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:22:39.495393 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:22:39.495431 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:39.495490 (+ 59us) server_negotiation.cc:299] Negotiation successful
0812 14:22:39.495509 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 14:22:39.495587 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:39.493142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:42551 (local address 127.5.148.3:54653)
0812 14:22:39.493323 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:39.493342 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:39.493389 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:39.493515 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:39.493516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:39.493517 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:22:39.493612 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:39.493614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.494229 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.494230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:39.494877 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:39.494879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.495221 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.495222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:39.495272 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:39.495287 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:22:39.495449 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:22:39.495451 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:22:39.495482 (+ 31us) client_negotiation.cc:240] Negotiation successful
0812 14:22:39.495518 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":110,"threads_started":1}
I20260812 14:22:39.498107 7940 raft_consensus.cc:1261] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Refusing update from remote peer bc914e93afcb41bb941daf7d391da6eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:39.498134 8099 raft_consensus.cc:1261] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Refusing update from remote peer bc914e93afcb41bb941daf7d391da6eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:39.498396 8447 consensus_queue.cc:1035] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:39.498585 8454 consensus_queue.cc:1035] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:39.506390 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:39.504059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:59181 (local address 127.5.148.3:46553)
0812 14:22:39.504102 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:39.504104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:39.504141 (+ 37us) server_negotiation.cc:407] Connection header received
0812 14:22:39.504214 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:39.504215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:39.504228 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:39.504274 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:22:39.504405 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.504894 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.505656 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.505875 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.505955 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:39.506107 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:22:39.506143 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:22:39.506209 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:39.506277 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 14:22:39.506300 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:22:39.506392 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:39.503916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:46553 (local address 127.5.148.1:59181)
0812 14:22:39.504066 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:39.504079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:39.504154 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:39.504286 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:39.504287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:39.504289 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:22:39.504343 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:39.504344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.504983 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.504984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:39.505565 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:39.505567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.505955 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.505956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:39.506019 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:39.506024 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:22:39.506223 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:22:39.506225 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:22:39.506263 (+ 38us) client_negotiation.cc:240] Negotiation successful
0812 14:22:39.506313 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260812 14:22:39.513410 7747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:39.513551 7747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:39.514273 7747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d: Sending DeleteTablet for 3 replicas of tablet 45c65f79fcb847f48fc93d6a6b2ff506
I20260812 14:22:39.514561 8233 tablet_service.cc:1514] Processing DeleteTablet for tablet 45c65f79fcb847f48fc93d6a6b2ff506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:39 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:52258
I20260812 14:22:39.514590 8074 tablet_service.cc:1514] Processing DeleteTablet for tablet 45c65f79fcb847f48fc93d6a6b2ff506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:39 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:52058
I20260812 14:22:39.514680 8430 tablet_replica.cc:330] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666: stopping tablet replica
I20260812 14:22:39.514709 8429 tablet_replica.cc:330] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb: stopping tablet replica
I20260812 14:22:39.514739 8430 raft_consensus.cc:2229] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:39.514777 8430 raft_consensus.cc:2258] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:39.514786 8429 raft_consensus.cc:2229] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:22:39.514880 8429 raft_consensus.cc:2258] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:39.515195 8429 ts_tablet_manager.cc:1905] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:39.515234 7916 tablet_service.cc:1514] Processing DeleteTablet for tablet 45c65f79fcb847f48fc93d6a6b2ff506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:39 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:33778
I20260812 14:22:39.515206 7747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:22:39.515355 7747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:22:39.515539 8430 ts_tablet_manager.cc:1905] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:39.516831 8432 tablet_replica.cc:330] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc: stopping tablet replica
I20260812 14:22:39.516901 8432 raft_consensus.cc:2229] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:39.516935 8432 raft_consensus.cc:2258] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:39.517237 8429 ts_tablet_manager.cc:1918] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:22:39.517314 8429 log.cc:1196] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/45c65f79fcb847f48fc93d6a6b2ff506
I20260812 14:22:39.517453 8432 ts_tablet_manager.cc:1905] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:39.517557 8429 ts_tablet_manager.cc:1939] T 45c65f79fcb847f48fc93d6a6b2ff506 P bc914e93afcb41bb941daf7d391da6eb: Deleting consensus metadata
I20260812 14:22:39.517786 8233 tablet_service.cc:1467] Processing CreateTablet for tablet 6013915f77954a83a68ede5763c537a8 (DEFAULT_TABLE table=test-table [id=a511b455edaf414d80436382e774ed07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:39.517894 8233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6013915f77954a83a68ede5763c537a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:39.518884 8440 tablet_bootstrap.cc:492] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb: Bootstrap starting.
I20260812 14:22:39.518909 7735 catalog_manager.cc:4841] TS bc914e93afcb41bb941daf7d391da6eb (127.5.148.3:46553): tablet 45c65f79fcb847f48fc93d6a6b2ff506 (table test-table [id=19a50f54b07c4970a609f633d058e1e1]) successfully deleted
I20260812 14:22:39.519212 8432 ts_tablet_manager.cc:1918] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:22:39.519277 8432 log.cc:1196] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-0/wal/wals/45c65f79fcb847f48fc93d6a6b2ff506
I20260812 14:22:39.519259 7916 tablet_service.cc:1467] Processing CreateTablet for tablet 6013915f77954a83a68ede5763c537a8 (DEFAULT_TABLE table=test-table [id=a511b455edaf414d80436382e774ed07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:39.519346 8440 tablet_bootstrap.cc:654] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:39.519356 7916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6013915f77954a83a68ede5763c537a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:39.519513 8432 ts_tablet_manager.cc:1939] T 45c65f79fcb847f48fc93d6a6b2ff506 P 5a39686ed013474ea71811ea9eaf7cdc: Deleting consensus metadata
I20260812 14:22:39.520032 8440 tablet_bootstrap.cc:492] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb: No bootstrap required, opened a new log
I20260812 14:22:39.520107 8440 ts_tablet_manager.cc:1397] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:39.520300 8440 raft_consensus.cc:348] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.520388 8440 raft_consensus.cc:374] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:39.520421 8440 raft_consensus.cc:729] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc914e93afcb41bb941daf7d391da6eb, State: Initialized, Role: FOLLOWER
I20260812 14:22:39.520458 8440 consensus_queue.cc:260] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.520574 8440 ts_tablet_manager.cc:1428] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:39.520658 8443 tablet_bootstrap.cc:492] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc: Bootstrap starting.
I20260812 14:22:39.520709 8074 tablet_service.cc:1467] Processing CreateTablet for tablet 6013915f77954a83a68ede5763c537a8 (DEFAULT_TABLE table=test-table [id=a511b455edaf414d80436382e774ed07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:39.520924 7732 catalog_manager.cc:4841] TS 5a39686ed013474ea71811ea9eaf7cdc (127.5.148.1:42551): tablet 45c65f79fcb847f48fc93d6a6b2ff506 (table test-table [id=19a50f54b07c4970a609f633d058e1e1]) successfully deleted
I20260812 14:22:39.521095 8443 tablet_bootstrap.cc:654] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:39.521138 8430 ts_tablet_manager.cc:1918] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:22:39.521145 8074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6013915f77954a83a68ede5763c537a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:39.521210 8430 log.cc:1196] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/45c65f79fcb847f48fc93d6a6b2ff506
I20260812 14:22:39.521476 8430 ts_tablet_manager.cc:1939] T 45c65f79fcb847f48fc93d6a6b2ff506 P 70e3ab9792e34f8cbab1136cdf719666: Deleting consensus metadata
I20260812 14:22:39.521736 8443 tablet_bootstrap.cc:492] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc: No bootstrap required, opened a new log
I20260812 14:22:39.521783 8443 ts_tablet_manager.cc:1397] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:39.521806 7732 catalog_manager.cc:4841] TS 70e3ab9792e34f8cbab1136cdf719666 (127.5.148.2:41513): tablet 45c65f79fcb847f48fc93d6a6b2ff506 (table test-table [id=19a50f54b07c4970a609f633d058e1e1]) successfully deleted
I20260812 14:22:39.522040 8443 raft_consensus.cc:348] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.522137 8443 raft_consensus.cc:374] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:39.522162 8443 raft_consensus.cc:729] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a39686ed013474ea71811ea9eaf7cdc, State: Initialized, Role: FOLLOWER
I20260812 14:22:39.522205 8443 consensus_queue.cc:260] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.522300 8443 ts_tablet_manager.cc:1428] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:39.522715 8442 tablet_bootstrap.cc:492] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666: Bootstrap starting.
I20260812 14:22:39.523077 8442 tablet_bootstrap.cc:654] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:39.523705 8442 tablet_bootstrap.cc:492] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666: No bootstrap required, opened a new log
I20260812 14:22:39.523777 8442 ts_tablet_manager.cc:1397] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:39.523942 8442 raft_consensus.cc:348] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.524020 8442 raft_consensus.cc:374] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:39.524045 8442 raft_consensus.cc:729] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e3ab9792e34f8cbab1136cdf719666, State: Initialized, Role: FOLLOWER
I20260812 14:22:39.524093 8442 consensus_queue.cc:260] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.524217 8442 ts_tablet_manager.cc:1428] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:22:39.562739 8404 raft_consensus.cc:482] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:22:39.562894 8404 raft_consensus.cc:504] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.563122 8404 leader_election.cc:288] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70e3ab9792e34f8cbab1136cdf719666 (127.5.148.2:41513), bc914e93afcb41bb941daf7d391da6eb (127.5.148.3:46553)
I20260812 14:22:39.563341 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6013915f77954a83a68ede5763c537a8" candidate_uuid: "5a39686ed013474ea71811ea9eaf7cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e3ab9792e34f8cbab1136cdf719666" is_pre_election: true
I20260812 14:22:39.563390 8257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6013915f77954a83a68ede5763c537a8" candidate_uuid: "5a39686ed013474ea71811ea9eaf7cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc914e93afcb41bb941daf7d391da6eb" is_pre_election: true
I20260812 14:22:39.563473 8099 raft_consensus.cc:2454] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a39686ed013474ea71811ea9eaf7cdc in term 0.
I20260812 14:22:39.563500 8257 raft_consensus.cc:2454] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a39686ed013474ea71811ea9eaf7cdc in term 0.
I20260812 14:22:39.563652 7857 leader_election.cc:302] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a39686ed013474ea71811ea9eaf7cdc, 70e3ab9792e34f8cbab1136cdf719666; no voters:
I20260812 14:22:39.563756 8404 raft_consensus.cc:2781] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:22:39.563792 8404 raft_consensus.cc:482] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:22:39.563823 8404 raft_consensus.cc:3037] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:39.564795 8404 raft_consensus.cc:504] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.564958 8404 leader_election.cc:288] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [CANDIDATE]: Term 1 election: Requested vote from peers 70e3ab9792e34f8cbab1136cdf719666 (127.5.148.2:41513), bc914e93afcb41bb941daf7d391da6eb (127.5.148.3:46553)
I20260812 14:22:39.565186 8257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6013915f77954a83a68ede5763c537a8" candidate_uuid: "5a39686ed013474ea71811ea9eaf7cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc914e93afcb41bb941daf7d391da6eb"
I20260812 14:22:39.565176 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6013915f77954a83a68ede5763c537a8" candidate_uuid: "5a39686ed013474ea71811ea9eaf7cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e3ab9792e34f8cbab1136cdf719666"
I20260812 14:22:39.565266 8257 raft_consensus.cc:3037] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:39.565266 8099 raft_consensus.cc:3037] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:39.566064 8257 raft_consensus.cc:2454] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a39686ed013474ea71811ea9eaf7cdc in term 1.
I20260812 14:22:39.566064 8099 raft_consensus.cc:2454] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a39686ed013474ea71811ea9eaf7cdc in term 1.
I20260812 14:22:39.566239 7857 leader_election.cc:302] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a39686ed013474ea71811ea9eaf7cdc, 70e3ab9792e34f8cbab1136cdf719666; no voters:
I20260812 14:22:39.566375 8404 raft_consensus.cc:2781] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:39.566464 8404 raft_consensus.cc:686] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [term 1 LEADER]: Becoming Leader. State: Replica: 5a39686ed013474ea71811ea9eaf7cdc, State: Running, Role: LEADER
I20260812 14:22:39.566525 8404 consensus_queue.cc:237] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } }
I20260812 14:22:39.567162 7749 catalog_manager.cc:5495] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a39686ed013474ea71811ea9eaf7cdc (127.5.148.1). New cstate: current_term: 1 leader_uuid: "5a39686ed013474ea71811ea9eaf7cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a39686ed013474ea71811ea9eaf7cdc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 42551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 } health_report { overall_health: UNKNOWN } } }
I20260812 14:22:39.583386 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:39.580779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.5.148.1:42551)
0812 14:22:39.580828 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:39.580830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:39.580841 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:22:39.580896 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:39.580897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:39.580907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:39.580957 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:22:39.581085 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.581566 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.582648 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.582863 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.582942 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:39.583093 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:22:39.583123 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:22:39.583164 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:39.583230 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 14:22:39.583254 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:22:39.588729 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:39.586055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:42551 (local address 127.5.148.1:42791)
0812 14:22:39.586115 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:39.586127 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:39.586168 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:39.586355 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:39.586356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:39.586357 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:22:39.586395 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:39.586396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.587240 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.587241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:39.587792 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:39.587794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.588304 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.588305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:39.588377 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:39.588383 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:22:39.588576 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:22:39.588578 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:22:39.588613 (+ 35us) client_negotiation.cc:240] Negotiation successful
0812 14:22:39.588652 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260812 14:22:39.588737 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:39.586191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:42791 (local address 127.5.148.1:42551)
0812 14:22:39.586249 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:39.586251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:39.586259 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:22:39.586281 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:39.586282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:39.586292 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:39.586354 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:22:39.586458 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.587130 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.587884 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:39.588222 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:39.588297 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:39.588467 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:22:39.588508 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:22:39.588570 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:39.588619 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 14:22:39.588640 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:22:39.591195 8257 raft_consensus.cc:1261] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Refusing update from remote peer 5a39686ed013474ea71811ea9eaf7cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:39.591435 8099 raft_consensus.cc:1261] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Refusing update from remote peer 5a39686ed013474ea71811ea9eaf7cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:39.591507 8404 consensus_queue.cc:1035] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc914e93afcb41bb941daf7d391da6eb" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:39.591780 8404 consensus_queue.cc:1035] T 6013915f77954a83a68ede5763c537a8 P 5a39686ed013474ea71811ea9eaf7cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "70e3ab9792e34f8cbab1136cdf719666" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 14:22:39.597071 8401 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 14:22:41.118079 7747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:41.118216 7747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:41.119202 7747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b623f6c38ae34534af26e792b1c9083d: Sending DeleteTablet for 3 replicas of tablet 6013915f77954a83a68ede5763c537a8
I20260812 14:22:41.119517 7916 tablet_service.cc:1514] Processing DeleteTablet for tablet 6013915f77954a83a68ede5763c537a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:41 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:33778
I20260812 14:22:41.119534 8233 tablet_service.cc:1514] Processing DeleteTablet for tablet 6013915f77954a83a68ede5763c537a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:41 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:52258
I20260812 14:22:41.119555 8074 tablet_service.cc:1514] Processing DeleteTablet for tablet 6013915f77954a83a68ede5763c537a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:41 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:52058
I20260812 14:22:41.119664 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 7837
I20260812 14:22:41.131181 8498 tablet_replica.cc:330] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666: stopping tablet replica
I20260812 14:22:41.131167 8500 tablet_replica.cc:330] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb: stopping tablet replica
I20260812 14:22:41.131271 8498 raft_consensus.cc:2229] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:41.131289 8500 raft_consensus.cc:2229] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:41.131338 8498 raft_consensus.cc:2258] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:41.131371 8500 raft_consensus.cc:2258] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:41.131665 8498 ts_tablet_manager.cc:1905] T 6013915f77954a83a68ede5763c537a8 P 70e3ab9792e34f8cbab1136cdf719666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:41.131742 8500 ts_tablet_manager.cc:1905] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 14:22:41.132617 7732 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:42551 (error 108)
I20260812 14:22:41.132997 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 8002
I20260812 14:22:41.134348 8500 ts_tablet_manager.cc:1918] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 14:22:41.134441 8500 log.cc:1196] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/6013915f77954a83a68ede5763c537a8
I20260812 14:22:41.134712 8500 ts_tablet_manager.cc:1939] T 6013915f77954a83a68ede5763c537a8 P bc914e93afcb41bb941daf7d391da6eb: Deleting consensus metadata
I20260812 14:22:41.135272 7735 catalog_manager.cc:4841] TS bc914e93afcb41bb941daf7d391da6eb (127.5.148.3:46553): tablet 6013915f77954a83a68ede5763c537a8 (table test-table [id=a511b455edaf414d80436382e774ed07]) successfully deleted
I20260812 14:22:41.135596 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:41.133166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:42551 (local address 127.0.0.1:33806)
0812 14:22:41.135479 (+ 2313us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:41.135540 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2280,"thread_start_us":69,"threads_started":1}
W20260812 14:22:41.135740 7732 catalog_manager.cc:4584] TS 5a39686ed013474ea71811ea9eaf7cdc (127.5.148.1:42551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6013915f77954a83a68ede5763c537a8: Network error: Client connection negotiation failed: client connection to 127.5.148.1:42551: connect: Connection refused (error 111)
I20260812 14:22:41.143066 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 8161
I20260812 14:22:41.144591 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:41.143079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:41513 (local address 127.0.0.1:52078)
0812 14:22:41.143168 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:41.143211 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.2:41513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 14:22:41.155066 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 7713
2026-08-12T14:22:41Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6133 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:22:41 dist-test-slave-8w9v krb5kdc[8505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:22:41 dist-test-slave-8w9v krb5kdc[8505](info): set up 2 sockets
Aug 12 14:22:41 dist-test-slave-8w9v krb5kdc[8505](info): commencing operation
krb5kdc: starting...
W20260812 14:22:42.477033 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.295s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:22:42 dist-test-slave-8w9v krb5kdc[8505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:22:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:22:42.742328 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:45209
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:45209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:42.982738 8523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:42.983566 8523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:42.983697 8523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:42.988047 8523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:42.988205 8523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:42.988269 8523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:42.988312 8523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:42.988336 8523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:22:42.993743 8523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:45209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:45209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:42.994920 8523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:42.995796 8523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:43.008381 8532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:43.008605 8523 server_base.cc:1034] running on GCE node
W20260812 14:22:43.008397 8530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:43.008404 8529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:43.009601 8523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:43.010510 8523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:43.011888 8523 hybrid_clock.cc:648] HybridClock initialized: now 1786544563011823 us; error 113 us; skew 500 ppm
Aug 12 14:22:43 dist-test-slave-8w9v krb5kdc[8505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544563, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:43.026990 8523 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:22:43.029162 8523 webserver.cc:460] Webserver started at http://127.5.148.62:45045/ using document root <none> and password file <none>
I20260812 14:22:43.030000 8523 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:43.030282 8523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:43.030866 8523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:43.034478 8523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "e3c82afb8c9e496cabc23e0ca310e1cf"
format_stamp: "Formatted at 2026-08-12 14:22:43 on dist-test-slave-8w9v"
server_key: "4513b343aa5a45de06f9d03b8692af9b"
server_key_iv: "8452bdb374300803831b659356df4101"
server_key_version: "encryptionkey@0"
I20260812 14:22:43.035629 8523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "e3c82afb8c9e496cabc23e0ca310e1cf"
format_stamp: "Formatted at 2026-08-12 14:22:43 on dist-test-slave-8w9v"
server_key: "4513b343aa5a45de06f9d03b8692af9b"
server_key_iv: "8452bdb374300803831b659356df4101"
server_key_version: "encryptionkey@0"
I20260812 14:22:43.041082 8523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 14:22:43.043919 8541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:43.044476 8523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 14:22:43.044859 8523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "e3c82afb8c9e496cabc23e0ca310e1cf"
format_stamp: "Formatted at 2026-08-12 14:22:43 on dist-test-slave-8w9v"
server_key: "4513b343aa5a45de06f9d03b8692af9b"
server_key_iv: "8452bdb374300803831b659356df4101"
server_key_version: "encryptionkey@0"
I20260812 14:22:43.045053 8523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:43.073316 8523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:43.080348 8523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:43.080955 8523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:43.095225 8523 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:45209
I20260812 14:22:43.095412 8604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:45209 every 8 connection(s)
I20260812 14:22:43.096684 8523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:43.098568 8606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:43.103053 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 8523
I20260812 14:22:43.103292 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:43.103818 5712 external_mini_cluster.cc:1428] Setting key 6f39996980706ff42cd3fa11acb885b1
I20260812 14:22:43.106215 8606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf: Bootstrap starting.
I20260812 14:22:43.108045 8606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:43.108816 8606 log.cc:824] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:43.110484 8606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf: No bootstrap required, opened a new log
I20260812 14:22:43.115509 8606 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c82afb8c9e496cabc23e0ca310e1cf" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 45209 } }
I20260812 14:22:43.116065 8606 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:43.116178 8606 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c82afb8c9e496cabc23e0ca310e1cf, State: Initialized, Role: FOLLOWER
I20260812 14:22:43.116454 8606 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c82afb8c9e496cabc23e0ca310e1cf" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 45209 } }
I20260812 14:22:43.116597 8606 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:43.116649 8606 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:43.116778 8606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [term 0 FOLLOWER]: Advancing to term 1
Aug 12 14:22:43 dist-test-slave-8w9v krb5kdc[8505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:43.118127 8606 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c82afb8c9e496cabc23e0ca310e1cf" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 45209 } }
I20260812 14:22:43.118371 8606 leader_election.cc:302] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3c82afb8c9e496cabc23e0ca310e1cf; no voters:
I20260812 14:22:43.118767 8606 leader_election.cc:288] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:43.118804 8611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:43.119232 8611 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [term 1 LEADER]: Becoming Leader. State: Replica: e3c82afb8c9e496cabc23e0ca310e1cf, State: Running, Role: LEADER
I20260812 14:22:43.119635 8606 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:43.119616 8611 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c82afb8c9e496cabc23e0ca310e1cf" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 45209 } }
I20260812 14:22:43.121027 8613 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3c82afb8c9e496cabc23e0ca310e1cf. Latest consensus state: current_term: 1 leader_uuid: "e3c82afb8c9e496cabc23e0ca310e1cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c82afb8c9e496cabc23e0ca310e1cf" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 45209 } } }
I20260812 14:22:43.121268 8613 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:43.121716 8614 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3c82afb8c9e496cabc23e0ca310e1cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c82afb8c9e496cabc23e0ca310e1cf" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 45209 } } }
I20260812 14:22:43.121973 8614 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:43.123499 8621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:22:43.124708 8621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:22:43.127694 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:43.105061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46738 (local address 127.5.148.62:45209)
0812 14:22:43.105339 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:43.105357 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:43.105399 (+ 42us) server_negotiation.cc:407] Connection header received
0812 14:22:43.106226 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:43.106241 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:43.106398 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:43.106661 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:43.108723 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:43.110796 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:43.112717 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:43.113572 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:43.113730 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:43.119099 (+ 5369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:43.119109 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:43.119110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:43.119126 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:43.124244 (+ 5118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:43.125086 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:43.125091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:43.125091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:43.125249 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:43.125688 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:43.125689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:43.125689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:43.126161 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:43.126526 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:43.126741 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 14:22:43.127309 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":130,"threads_started":1}
I20260812 14:22:43.128394 8621 catalog_manager.cc:1347] Generated new cluster ID: 3641b219d80c40499eff138bb6eaec19
I20260812 14:22:43.128494 8621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:22:43.194407 8621 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:43.195622 8621 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:43.212011 8621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3c82afb8c9e496cabc23e0ca310e1cf: Generated new TSK 0
I20260812 14:22:43.212622 8621 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:22:43.272842 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:45209
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:43.462459 8637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:43.462746 8637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:43.462780 8637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:43.464334 8637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:43.464391 8637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:43.464425 8637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:22:43.466261 8637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:45209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:43.466601 8637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:43.466912 8637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:43.473114 8646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:43.473174 8644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:43.473114 8642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:43.473397 8637 server_base.cc:1034] running on GCE node
I20260812 14:22:43.473676 8637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:43.473954 8637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:43.475122 8637 hybrid_clock.cc:648] HybridClock initialized: now 1786544563475105 us; error 28 us; skew 500 ppm
Aug 12 14:22:43 dist-test-slave-8w9v krb5kdc[8505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544563, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:43.478469 8637 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:22:43.479072 8637 webserver.cc:460] Webserver started at http://127.5.148.1:44623/ using document root <none> and password file <none>
I20260812 14:22:43.479372 8637 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:43.479432 8637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:43.479605 8637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:43.480944 8637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "305a5ec6d21748cbbb38b960a2f2f2fd"
format_stamp: "Formatted at 2026-08-12 14:22:43 on dist-test-slave-8w9v"
server_key: "537264daee60394bf28d6247921b7bb3"
server_key_iv: "de9e1a424863bd1752acf57ad309ef5a"
server_key_version: "encryptionkey@0"
I20260812 14:22:43.481377 8637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "305a5ec6d21748cbbb38b960a2f2f2fd"
format_stamp: "Formatted at 2026-08-12 14:22:43 on dist-test-slave-8w9v"
server_key: "537264daee60394bf28d6247921b7bb3"
server_key_iv: "de9e1a424863bd1752acf57ad309ef5a"
server_key_version: "encryptionkey@0"
I20260812 14:22:43.483026 8637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:22:43.483991 8654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:43.484268 8637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:43.484432 8637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "305a5ec6d21748cbbb38b960a2f2f2fd"
format_stamp: "Formatted at 2026-08-12 14:22:43 on dist-test-slave-8w9v"
server_key: "537264daee60394bf28d6247921b7bb3"
server_key_iv: "de9e1a424863bd1752acf57ad309ef5a"
server_key_version: "encryptionkey@0"
I20260812 14:22:43.484517 8637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:43.506692 8637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:43.509001 8637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:43.509166 8637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:43.509481 8637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:43.509841 8637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:43.509898 8637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:43.509943 8637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:43.509986 8637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:43.517540 8637 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:39913
I20260812 14:22:43.517593 8788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:39913 every 8 connection(s)
I20260812 14:22:43.517964 8637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:22:43 dist-test-slave-8w9v krb5kdc[8505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544563, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:43.527858 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 8637
I20260812 14:22:43.527952 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:22:43.528157 5712 external_mini_cluster.cc:1428] Setting key 79584ef0c44a1361d8a7486db8315199
I20260812 14:22:43.528802 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:43.518824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:54393 (local address 127.5.148.62:45209)
0812 14:22:43.518945 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:43.518948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:43.519155 (+ 207us) server_negotiation.cc:407] Connection header received
0812 14:22:43.520430 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:43.520431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:43.520443 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:43.520529 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:43.521231 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:43.521750 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:43.522800 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:43.523009 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:43.523091 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:43.525554 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:43.525556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:43.525556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:43.525563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:43.526922 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:43.527403 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:43.527404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:43.527405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:43.527468 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:43.527811 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:43.527812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:43.527812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:43.528113 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:43.528264 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:43.528639 (+ 375us) server_negotiation.cc:299] Negotiation successful
0812 14:22:43.528735 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 14:22:43.529238 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:43.518773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:45209 (local address 127.5.148.1:54393)
0812 14:22:43.519023 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:43.519065 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:43.520285 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:43.520610 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:43.520613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:43.520941 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:43.521151 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:43.521155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:43.521848 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:43.521849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:43.522687 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:43.522689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:43.523093 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:43.523094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:43.523199 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:43.523767 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:43.523777 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:43.525434 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:43.527004 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:43.527011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:43.527013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:43.527304 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:43.527537 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:43.527538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:43.527538 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:43.527707 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:43.528292 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:43.528301 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:43.528476 (+ 175us) client_negotiation.cc:769] Sending connection context
0812 14:22:43.528618 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 14:22:43.528823 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":147,"threads_started":1}
I20260812 14:22:43.530014 8789 heartbeater.cc:347] Connected to a master server at 127.5.148.62:45209
I20260812 14:22:43.530136 8789 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:43.530359 8789 heartbeater.cc:511] Master 127.5.148.62:45209 requested a full tablet report, sending...
I20260812 14:22:43.530962 8559 ts_manager.cc:194] Registered new tserver with Master: 305a5ec6d21748cbbb38b960a2f2f2fd (127.5.148.1:39913)
I20260812 14:22:43.531679 8559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:54393
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:22:43.560027 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:45209
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:43.655011 8795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:43.655247 8795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:43.655282 8795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:43.656543 8795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:43.656590 8795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:43.656621 8795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:22:43.658115 8795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:45209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:43.658412 8795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:43.658659 8795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:43.664167 8801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:43.664167 8800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:43.664202 8803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:43.664417 8795 server_base.cc:1034] running on GCE node
I20260812 14:22:43.664636 8795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:43.664914 8795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:43.666141 8795 hybrid_clock.cc:648] HybridClock initialized: now 1786544563666131 us; error 36 us; skew 500 ppm
Aug 12 14:22:43 dist-test-slave-8w9v krb5kdc[8505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544563, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:43.669003 8795 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:22:43.669490 8795 webserver.cc:460] Webserver started at http://127.5.148.2:43919/ using document root <none> and password file <none>
I20260812 14:22:43.669737 8795 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:43.669786 8795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:43.669929 8795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:43.671059 8795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "2b4558b63c0947a5a8f7d5d46ab01410"
format_stamp: "Formatted at 2026-08-12 14:22:43 on dist-test-slave-8w9v"
server_key: "c62c590d64adf7915b6f11ea05222b61"
server_key_iv: "443e4b55697e005bedcde21e78ff31c9"
server_key_version: "encryptionkey@0"
I20260812 14:22:43.671445 8795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "2b4558b63c0947a5a8f7d5d46ab01410"
format_stamp: "Formatted at 2026-08-12 14:22:43 on dist-test-slave-8w9v"
server_key: "c62c590d64adf7915b6f11ea05222b61"
server_key_iv: "443e4b55697e005bedcde21e78ff31c9"
server_key_version: "encryptionkey@0"
I20260812 14:22:43.672989 8795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260812 14:22:43.673830 8810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:43.674052 8795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:43.674166 8795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "2b4558b63c0947a5a8f7d5d46ab01410"
format_stamp: "Formatted at 2026-08-12 14:22:43 on dist-test-slave-8w9v"
server_key: "c62c590d64adf7915b6f11ea05222b61"
server_key_iv: "443e4b55697e005bedcde21e78ff31c9"
server_key_version: "encryptionkey@0"
I20260812 14:22:43.674244 8795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:43.694983 8795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:43.697494 8795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:43.697677 8795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:43.697952 8795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:43.698293 8795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:43.698348 8795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:43.698405 8795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:43.698433 8795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:43.705104 8795 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:34531
I20260812 14:22:43.705188 8942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:34531 every 8 connection(s)
I20260812 14:22:43.705566 8795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:22:43.706492 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 8795
I20260812 14:22:43.706600 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:22:43.706821 5712 external_mini_cluster.cc:1428] Setting key ec0673274e87ddbb71453bc02f08014b
Aug 12 14:22:43 dist-test-slave-8w9v krb5kdc[8505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544563, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
I20260812 14:22:43.717868 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:43.706432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:38157 (local address 127.5.148.62:45209)
0812 14:22:43.706512 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:43.706515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:43.706928 (+ 413us) server_negotiation.cc:407] Connection header received
0812 14:22:43.707674 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:43.707675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:43.707685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:43.707730 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:43.709182 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:43.709952 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:43.711093 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:43.711416 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:43.711498 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:43.714183 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:43.714185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:43.714186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:43.714196 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:43.716077 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:43.716584 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:43.716585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:43.716586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:43.716660 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:43.716982 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:43.716984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:43.716984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:43.717151 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:43.717226 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:43.717641 (+ 415us) server_negotiation.cc:299] Negotiation successful
0812 14:22:43.717742 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 14:22:43.718020 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:43.706412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:45209 (local address 127.5.148.2:38157)
0812 14:22:43.706740 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:43.706820 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:43.707564 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:43.707822 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:43.707826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:43.708751 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:43.709071 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:43.709078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:43.710071 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:43.710072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:43.710973 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:43.710976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:43.711553 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:43.711555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:43.711711 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:43.712441 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:43.712451 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:43.714069 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:43.716215 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:43.716223 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:43.716225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:43.716494 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:43.716757 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:43.716758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:43.716759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:43.716899 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:43.717229 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:43.717236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:43.717425 (+ 189us) client_negotiation.cc:769] Sending connection context
0812 14:22:43.717573 (+ 148us) client_negotiation.cc:240] Negotiation successful
0812 14:22:43.717766 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":125,"threads_started":1}
I20260812 14:22:43.718582 8943 heartbeater.cc:347] Connected to a master server at 127.5.148.62:45209
I20260812 14:22:43.718683 8943 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:43.718914 8943 heartbeater.cc:511] Master 127.5.148.62:45209 requested a full tablet report, sending...
I20260812 14:22:43.719403 8559 ts_manager.cc:194] Registered new tserver with Master: 2b4558b63c0947a5a8f7d5d46ab01410 (127.5.148.2:34531)
I20260812 14:22:43.719801 8559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:38157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:22:43.740465 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:45209
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:43.835696 8948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:43.835914 8948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:43.835939 8948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:43.837389 8948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:43.837452 8948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:43.837492 8948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:22:43.839010 8948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:45209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:43.839304 8948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:43.839557 8948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:43.844931 8955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:43.844954 8957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:43.845021 8954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:43.845167 8948 server_base.cc:1034] running on GCE node
I20260812 14:22:43.845376 8948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:43.845639 8948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:43.846817 8948 hybrid_clock.cc:648] HybridClock initialized: now 1786544563846796 us; error 31 us; skew 500 ppm
Aug 12 14:22:43 dist-test-slave-8w9v krb5kdc[8505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544563, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:43.849525 8948 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:22:43.850013 8948 webserver.cc:460] Webserver started at http://127.5.148.3:38715/ using document root <none> and password file <none>
I20260812 14:22:43.850266 8948 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:43.850335 8948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:43.850492 8948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:43.851619 8948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "a5a63e28eeea431e858e9fdc56a48577"
format_stamp: "Formatted at 2026-08-12 14:22:43 on dist-test-slave-8w9v"
server_key: "a9bbfbff31ef8ce776e50be56cfc131b"
server_key_iv: "05633c8b54e7f8de1e5261bb24824d75"
server_key_version: "encryptionkey@0"
I20260812 14:22:43.851997 8948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "a5a63e28eeea431e858e9fdc56a48577"
format_stamp: "Formatted at 2026-08-12 14:22:43 on dist-test-slave-8w9v"
server_key: "a9bbfbff31ef8ce776e50be56cfc131b"
server_key_iv: "05633c8b54e7f8de1e5261bb24824d75"
server_key_version: "encryptionkey@0"
I20260812 14:22:43.853451 8948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:43.854197 8966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:43.854373 8948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:22:43.854465 8948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "a5a63e28eeea431e858e9fdc56a48577"
format_stamp: "Formatted at 2026-08-12 14:22:43 on dist-test-slave-8w9v"
server_key: "a9bbfbff31ef8ce776e50be56cfc131b"
server_key_iv: "05633c8b54e7f8de1e5261bb24824d75"
server_key_version: "encryptionkey@0"
I20260812 14:22:43.854549 8948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:43.887578 8948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:43.890084 8948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:43.890308 8948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:43.890638 8948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:43.891026 8948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:43.891091 8948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:43.891142 8948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:43.891175 8948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:43.899201 8948 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:36625
I20260812 14:22:43.899256 9097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:36625 every 8 connection(s)
I20260812 14:22:43.899657 8948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:22:43 dist-test-slave-8w9v krb5kdc[8505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544563, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:43.908291 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 8948
I20260812 14:22:43.908411 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:22:43.908617 5712 external_mini_cluster.cc:1428] Setting key 8391d1d51bc5a6cd5ccf21cf46d63931
I20260812 14:22:43.910702 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:43.900540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:39257 (local address 127.5.148.62:45209)
0812 14:22:43.900596 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:43.900598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:43.900890 (+ 292us) server_negotiation.cc:407] Connection header received
0812 14:22:43.901567 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:43.901569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:43.901583 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:43.901652 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:43.902782 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:43.903303 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:43.904417 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:43.904627 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:43.904716 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:43.907249 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:43.907251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:43.907251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:43.907259 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:43.909042 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:43.909561 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:43.909562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:43.909563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:43.909634 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:43.909941 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:43.909942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:43.909942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:43.910086 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:43.910173 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:43.910494 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 14:22:43.910593 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:22:43.910970 9101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:43.900497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:45209 (local address 127.5.148.3:39257)
0812 14:22:43.900708 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:43.900776 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:43.901439 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:43.901732 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:43.901737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:43.902445 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:43.902665 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:43.902670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:43.903422 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:43.903424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:43.904316 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:43.904318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:43.904733 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:43.904734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:43.904885 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:43.905520 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:43.905529 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:43.907140 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:43.909149 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:43.909160 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:43.909163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:43.909453 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:43.909710 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:43.909711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:43.909712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:43.909848 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:43.910162 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:43.910171 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:43.910340 (+ 169us) client_negotiation.cc:769] Sending connection context
0812 14:22:43.910470 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 14:22:43.910678 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":107,"threads_started":1}
I20260812 14:22:43.911558 9099 heartbeater.cc:347] Connected to a master server at 127.5.148.62:45209
I20260812 14:22:43.911669 9099 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:43.911832 9099 heartbeater.cc:511] Master 127.5.148.62:45209 requested a full tablet report, sending...
I20260812 14:22:43.912299 8559 ts_manager.cc:194] Registered new tserver with Master: a5a63e28eeea431e858e9fdc56a48577 (127.5.148.3:36625)
I20260812 14:22:43.912662 8559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:39257
I20260812 14:22:43.920424 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 14:22:43.936600 8609 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:22:43.935334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49306 (local address 127.5.148.62:45209)
0812 14:22:43.935443 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:43.935446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:43.935570 (+ 124us) server_negotiation.cc:407] Connection header received
0812 14:22:43.935675 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:43.935676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:43.935689 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:43.935778 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:43.936517 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49306: BlockingRecv error: recv got EOF from 127.0.0.1:49306 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 14:22:43.938088 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 8637
I20260812 14:22:43.948336 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 8795
I20260812 14:22:43.959069 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 8948
I20260812 14:22:43.966941 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 8523
2026-08-12T14:22:43Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2810 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:22:44 dist-test-slave-8w9v krb5kdc[9113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:22:44 dist-test-slave-8w9v krb5kdc[9113](info): set up 2 sockets
Aug 12 14:22:44 dist-test-slave-8w9v krb5kdc[9113](info): commencing operation
krb5kdc: starting...
W20260812 14:22:45.089234 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:22:45 dist-test-slave-8w9v krb5kdc[9113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:22:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:22:45.223452 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:46479
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:46479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:45.308300 9127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:45.308516 9127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:45.308540 9127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:45.309723 9127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:45.309769 9127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:45.309784 9127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:45.309796 9127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:45.309809 9127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:22:45.311352 9127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:46479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:46479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:45.311620 9127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:45.311861 9127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:22:45.317030 9127 server_base.cc:1034] running on GCE node
W20260812 14:22:45.317000 9132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:45.317054 9135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:45.317000 9133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:45.317477 9127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:45.317790 9127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:45.318979 9127 hybrid_clock.cc:648] HybridClock initialized: now 1786544565318973 us; error 34 us; skew 500 ppm
Aug 12 14:22:45 dist-test-slave-8w9v krb5kdc[9113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544565, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:45.321659 9127 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:22:45.322111 9127 webserver.cc:460] Webserver started at http://127.5.148.62:41749/ using document root <none> and password file <none>
I20260812 14:22:45.322366 9127 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:45.322414 9127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:45.322547 9127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:45.323647 9127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "1a6653e008624b16a020dc4a07d1d356"
format_stamp: "Formatted at 2026-08-12 14:22:45 on dist-test-slave-8w9v"
server_key: "ba3511584362e6cbd668975939479d28"
server_key_iv: "a0b475b6c1518de6d34be58447ef4d43"
server_key_version: "encryptionkey@0"
I20260812 14:22:45.324024 9127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "1a6653e008624b16a020dc4a07d1d356"
format_stamp: "Formatted at 2026-08-12 14:22:45 on dist-test-slave-8w9v"
server_key: "ba3511584362e6cbd668975939479d28"
server_key_iv: "a0b475b6c1518de6d34be58447ef4d43"
server_key_version: "encryptionkey@0"
I20260812 14:22:45.325434 9127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:45.326159 9145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:45.326345 9127 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:22:45.326457 9127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "1a6653e008624b16a020dc4a07d1d356"
format_stamp: "Formatted at 2026-08-12 14:22:45 on dist-test-slave-8w9v"
server_key: "ba3511584362e6cbd668975939479d28"
server_key_iv: "a0b475b6c1518de6d34be58447ef4d43"
server_key_version: "encryptionkey@0"
I20260812 14:22:45.326541 9127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:45.337983 9127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:45.340549 9127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:45.340734 9127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:45.344914 9127 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:46479
I20260812 14:22:45.344985 9203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:46479 every 8 connection(s)
I20260812 14:22:45.345376 9127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:45.346382 9204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:45.348924 9204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356: Bootstrap starting.
I20260812 14:22:45.349617 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 9127
I20260812 14:22:45.349645 9204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:45.349730 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:45.349947 5712 external_mini_cluster.cc:1428] Setting key 901f3b726948cce1fc42bd73136db702
I20260812 14:22:45.349997 9204 log.cc:824] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:45.350867 9204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356: No bootstrap required, opened a new log
I20260812 14:22:45.353500 9204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a6653e008624b16a020dc4a07d1d356" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46479 } }
I20260812 14:22:45.353673 9204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:45.353708 9204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a6653e008624b16a020dc4a07d1d356, State: Initialized, Role: FOLLOWER
I20260812 14:22:45.353863 9204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a6653e008624b16a020dc4a07d1d356" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46479 } }
I20260812 14:22:45.353977 9204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:45.354022 9204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:45.354074 9204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:45.354696 9204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a6653e008624b16a020dc4a07d1d356" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46479 } }
I20260812 14:22:45.354806 9204 leader_election.cc:302] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a6653e008624b16a020dc4a07d1d356; no voters:
I20260812 14:22:45.354955 9204 leader_election.cc:288] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:45.355020 9209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:45.355185 9209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [term 1 LEADER]: Becoming Leader. State: Replica: 1a6653e008624b16a020dc4a07d1d356, State: Running, Role: LEADER
I20260812 14:22:45.355209 9204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:45.356258 9209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a6653e008624b16a020dc4a07d1d356" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46479 } }
I20260812 14:22:45.356938 9212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a6653e008624b16a020dc4a07d1d356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a6653e008624b16a020dc4a07d1d356" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46479 } } }
I20260812 14:22:45.357098 9210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a6653e008624b16a020dc4a07d1d356. Latest consensus state: current_term: 1 leader_uuid: "1a6653e008624b16a020dc4a07d1d356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a6653e008624b16a020dc4a07d1d356" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46479 } } }
W20260812 14:22:45.357240 9224 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260812 14:22:45.357548 9212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:45.357564 9210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356 [sys.catalog]: This master's current role is: LEADER
W20260812 14:22:45.357630 9224 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:22:45.357769 9227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:22:45.357961 9227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 14:22:45 dist-test-slave-8w9v krb5kdc[9113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:45.359511 9227 catalog_manager.cc:1347] Generated new cluster ID: 13c5fcd1de234be59d9fc2387df8dfe5
I20260812 14:22:45.359572 9227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:22:45.363253 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:45.350539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.5.148.62:46479)
0812 14:22:45.350738 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:45.350748 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:45.350781 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:22:45.351381 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:45.351391 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:45.351458 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:45.351619 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:45.352313 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:45.353715 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:45.354597 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:45.355244 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:45.355391 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:45.358575 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:45.358579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:45.358579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:45.358592 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:45.360969 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:45.361479 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:45.361483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:45.361483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:45.361593 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:45.361825 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:45.361826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:45.361826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:45.362147 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:45.362797 (+ 650us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:45.362880 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 14:22:45.363027 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260812 14:22:45.364322 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:22:45.370306 9227 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:45.371026 9227 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:45.376925 9227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a6653e008624b16a020dc4a07d1d356: Generated new TSK 0
I20260812 14:22:45.377126 9227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:22:45.383374 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:45.379241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.5.148.62:46479)
0812 14:22:45.379308 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:45.379311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:45.379440 (+ 129us) server_negotiation.cc:407] Connection header received
0812 14:22:45.379523 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:45.379524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:45.379533 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:45.379586 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:45.380117 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:45.380762 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:45.381881 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:45.382082 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:45.382141 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:45.382883 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:45.382884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:45.382885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:45.383078 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:45.383096 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:45.383102 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:45.383152 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:45.383234 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 14:22:45.383263 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:22:45.386325 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 9127
2026-08-12T14:22:45Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1419 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:22:45 dist-test-slave-8w9v krb5kdc[9239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:22:45 dist-test-slave-8w9v krb5kdc[9239](info): set up 2 sockets
Aug 12 14:22:45 dist-test-slave-8w9v krb5kdc[9239](info): commencing operation
krb5kdc: starting...
W20260812 14:22:46.529791 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:22:46 dist-test-slave-8w9v krb5kdc[9239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:22:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:22:46.667835 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:38625
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:38625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:46.755450 9257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:46.755731 9257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:46.755774 9257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:46.757019 9257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:46.757098 9257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:46.757122 9257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:46.757143 9257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:46.757176 9257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:22:46.758738 9257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:38625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:38625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:46.759047 9257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:46.759294 9257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:46.764655 9268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:46.764755 9265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:46.764732 9257 server_base.cc:1034] running on GCE node
W20260812 14:22:46.764700 9264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:46.765290 9257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:46.765645 9257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:46.766799 9257 hybrid_clock.cc:648] HybridClock initialized: now 1786544566766772 us; error 50 us; skew 500 ppm
Aug 12 14:22:46 dist-test-slave-8w9v krb5kdc[9239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544566, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:46.769922 9257 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:22:46.770551 9257 webserver.cc:460] Webserver started at http://127.5.148.62:40831/ using document root <none> and password file <none>
I20260812 14:22:46.770964 9257 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:46.771040 9257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:46.771203 9257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:46.772543 9257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "8a6b0c2f952d4cb889519e08548a5967"
format_stamp: "Formatted at 2026-08-12 14:22:46 on dist-test-slave-8w9v"
server_key: "ab7ac7357901968267d68065a5afe6ab"
server_key_iv: "d4f697f722ecf87553ae85db8156e5c9"
server_key_version: "encryptionkey@0"
I20260812 14:22:46.772986 9257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "8a6b0c2f952d4cb889519e08548a5967"
format_stamp: "Formatted at 2026-08-12 14:22:46 on dist-test-slave-8w9v"
server_key: "ab7ac7357901968267d68065a5afe6ab"
server_key_iv: "d4f697f722ecf87553ae85db8156e5c9"
server_key_version: "encryptionkey@0"
I20260812 14:22:46.774547 9257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:22:46.775475 9276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:46.775712 9257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:46.775812 9257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "8a6b0c2f952d4cb889519e08548a5967"
format_stamp: "Formatted at 2026-08-12 14:22:46 on dist-test-slave-8w9v"
server_key: "ab7ac7357901968267d68065a5afe6ab"
server_key_iv: "d4f697f722ecf87553ae85db8156e5c9"
server_key_version: "encryptionkey@0"
I20260812 14:22:46.775878 9257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:46.788621 9257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:46.790870 9257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:46.791005 9257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:46.795524 9257 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:38625
I20260812 14:22:46.795599 9336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:38625 every 8 connection(s)
I20260812 14:22:46.796051 9257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:46.797482 9337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:46.799882 9337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967: Bootstrap starting.
I20260812 14:22:46.800527 9337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:46.800839 9337 log.cc:824] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:46.801663 9337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967: No bootstrap required, opened a new log
I20260812 14:22:46.803365 9337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6b0c2f952d4cb889519e08548a5967" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38625 } }
I20260812 14:22:46.803534 9337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:46.803560 9337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a6b0c2f952d4cb889519e08548a5967, State: Initialized, Role: FOLLOWER
I20260812 14:22:46.803740 9337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6b0c2f952d4cb889519e08548a5967" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38625 } }
I20260812 14:22:46.803813 9337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:46.803835 9337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:46.803892 9337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:46.804354 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 9257
I20260812 14:22:46.804495 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:46.804700 5712 external_mini_cluster.cc:1428] Setting key 8150ed1f532bbca84dfcaa4f8f85cc81
I20260812 14:22:46.805223 9337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6b0c2f952d4cb889519e08548a5967" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38625 } }
I20260812 14:22:46.805348 9337 leader_election.cc:302] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a6b0c2f952d4cb889519e08548a5967; no voters:
I20260812 14:22:46.805583 9337 leader_election.cc:288] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:46.805641 9342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:46.805852 9337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:46.805871 9342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [term 1 LEADER]: Becoming Leader. State: Replica: 8a6b0c2f952d4cb889519e08548a5967, State: Running, Role: LEADER
I20260812 14:22:46.806128 9342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6b0c2f952d4cb889519e08548a5967" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38625 } }
I20260812 14:22:46.806548 9345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a6b0c2f952d4cb889519e08548a5967. Latest consensus state: current_term: 1 leader_uuid: "8a6b0c2f952d4cb889519e08548a5967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6b0c2f952d4cb889519e08548a5967" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38625 } } }
I20260812 14:22:46.806529 9343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a6b0c2f952d4cb889519e08548a5967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6b0c2f952d4cb889519e08548a5967" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38625 } } }
I20260812 14:22:46.806756 9343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:46.806984 9353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:22:46.806982 9345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:46.807571 9353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:22:46.809355 9353 catalog_manager.cc:1347] Generated new cluster ID: d9a17c17b3154f1bae6467f2c7b04eb5
I20260812 14:22:46.809433 9353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:22:46 dist-test-slave-8w9v krb5kdc[9239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:46.816256 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:46.805229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.5.148.62:38625)
0812 14:22:46.805484 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:46.805495 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:46.805540 (+ 45us) server_negotiation.cc:407] Connection header received
0812 14:22:46.805673 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:46.805683 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:46.805748 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:46.806657 (+ 909us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:46.806850 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:46.808421 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:46.809406 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:46.809884 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:46.810023 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:46.812616 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:46.812621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:46.812621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:46.812635 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:46.814501 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:46.814907 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:46.814910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:46.814911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:46.815025 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:46.815246 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:46.815247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:46.815248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:46.815524 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:46.815711 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:46.815783 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 14:22:46.815949 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":198,"thread_start_us":156,"threads_started":1}
I20260812 14:22:46.853438 9353 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:46.854067 9353 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:46.871551 9353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a6b0c2f952d4cb889519e08548a5967: Generated new TSK 0
I20260812 14:22:46.871837 9353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:22:46.907780 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:38625
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:46.995976 9369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:46.996292 9369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:46.996338 9369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:46.997581 9369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:46.997658 9369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:46.997707 9369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:22:46.999230 9369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:38625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:46.999519 9369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:46.999800 9369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:47.005445 9377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:47.005425 9375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:47.005434 9374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:47.006187 9369 server_base.cc:1034] running on GCE node
I20260812 14:22:47.006404 9369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:47.006649 9369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:47.007800 9369 hybrid_clock.cc:648] HybridClock initialized: now 1786544567007774 us; error 42 us; skew 500 ppm
Aug 12 14:22:47 dist-test-slave-8w9v krb5kdc[9239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544567, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:47.010807 9369 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:22:47.011368 9369 webserver.cc:460] Webserver started at http://127.5.148.1:46303/ using document root <none> and password file <none>
I20260812 14:22:47.011636 9369 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:47.011709 9369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:47.011873 9369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:47.013108 9369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "0e68fc60edd642668d725431d47a9b13"
format_stamp: "Formatted at 2026-08-12 14:22:47 on dist-test-slave-8w9v"
server_key: "76b4272268b19cc07793cfdc60a7fb3f"
server_key_iv: "37b4cb3d3330fbe4d39fded3eb484bea"
server_key_version: "encryptionkey@0"
I20260812 14:22:47.013535 9369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "0e68fc60edd642668d725431d47a9b13"
format_stamp: "Formatted at 2026-08-12 14:22:47 on dist-test-slave-8w9v"
server_key: "76b4272268b19cc07793cfdc60a7fb3f"
server_key_iv: "37b4cb3d3330fbe4d39fded3eb484bea"
server_key_version: "encryptionkey@0"
I20260812 14:22:47.015133 9369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:22:47.016417 9385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:47.016673 9369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:47.016805 9369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "0e68fc60edd642668d725431d47a9b13"
format_stamp: "Formatted at 2026-08-12 14:22:47 on dist-test-slave-8w9v"
server_key: "76b4272268b19cc07793cfdc60a7fb3f"
server_key_iv: "37b4cb3d3330fbe4d39fded3eb484bea"
server_key_version: "encryptionkey@0"
I20260812 14:22:47.016899 9369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:47.030297 9369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:47.032907 9369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:47.033100 9369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:47.033380 9369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:47.033833 9369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:47.033892 9369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:47.033932 9369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:47.033964 9369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:47.042373 9369 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:38687
I20260812 14:22:47.042529 9516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:38687 every 8 connection(s)
I20260812 14:22:47.042944 9369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:22:47.044806 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 9369
I20260812 14:22:47.044924 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:22:47.045138 5712 external_mini_cluster.cc:1428] Setting key 5c9e0d08429bb6ea5db9e5f64a8dd115
Aug 12 14:22:47 dist-test-slave-8w9v krb5kdc[9239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544567, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
I20260812 14:22:47.060416 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:47.044038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:45493 (local address 127.5.148.62:38625)
0812 14:22:47.044095 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:47.044098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:47.049669 (+ 5571us) server_negotiation.cc:407] Connection header received
0812 14:22:47.050658 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:47.050660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:47.050675 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:47.050744 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:47.052028 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:47.052621 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:47.053794 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:47.054015 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:47.054106 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:47.056788 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:47.056789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:47.056789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:47.056797 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:47.058445 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:47.058875 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:47.058877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:47.058877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:47.058949 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:47.059448 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:47.059449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:47.059449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:47.059577 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:47.059643 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:47.060118 (+ 475us) server_negotiation.cc:299] Negotiation successful
0812 14:22:47.060270 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:22:47.060832 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:47.044156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38625 (local address 127.5.148.1:45493)
0812 14:22:47.049469 (+ 5313us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:47.049547 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:47.050516 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:47.050849 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:47.050855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:47.051659 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:47.051882 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:47.051887 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:47.052736 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:47.052737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:47.053687 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:47.053690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:47.054122 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:47.054124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:47.054271 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:47.054907 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:47.054926 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:47.056670 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:47.058532 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:47.058542 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:47.058544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:47.058789 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:47.059019 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:47.059020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:47.059021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:47.059182 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:47.059661 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:47.059672 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:47.059907 (+ 235us) client_negotiation.cc:769] Sending connection context
0812 14:22:47.060060 (+ 153us) client_negotiation.cc:240] Negotiation successful
0812 14:22:47.060321 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5238,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
I20260812 14:22:47.061499 9517 heartbeater.cc:347] Connected to a master server at 127.5.148.62:38625
I20260812 14:22:47.061635 9517 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:47.061863 9517 heartbeater.cc:511] Master 127.5.148.62:38625 requested a full tablet report, sending...
I20260812 14:22:47.062501 9297 ts_manager.cc:194] Registered new tserver with Master: 0e68fc60edd642668d725431d47a9b13 (127.5.148.1:38687)
I20260812 14:22:47.063141 9297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:45493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:22:47.079666 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:38625
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:47.179839 9523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:47.180104 9523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:47.180172 9523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:47.181501 9523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:47.181576 9523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:47.181624 9523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:22:47.183235 9523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:38625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:47.183547 9523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:47.183804 9523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:47.189518 9529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:47.189545 9528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:47.189642 9531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:47.189666 9523 server_base.cc:1034] running on GCE node
I20260812 14:22:47.190104 9523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:47.190380 9523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:47.191553 9523 hybrid_clock.cc:648] HybridClock initialized: now 1786544567191547 us; error 37 us; skew 500 ppm
Aug 12 14:22:47 dist-test-slave-8w9v krb5kdc[9239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544567, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:47.194303 9523 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:22:47.194778 9523 webserver.cc:460] Webserver started at http://127.5.148.2:41063/ using document root <none> and password file <none>
I20260812 14:22:47.195041 9523 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:47.195114 9523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:47.195266 9523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:47.196534 9523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "725d070008e943c6936a7fca45520879"
format_stamp: "Formatted at 2026-08-12 14:22:47 on dist-test-slave-8w9v"
server_key: "0e0ae7d746d48ec30fa8c5189d7de5f3"
server_key_iv: "0e5558036569187bb01e402bea3c9aeb"
server_key_version: "encryptionkey@0"
I20260812 14:22:47.196924 9523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "725d070008e943c6936a7fca45520879"
format_stamp: "Formatted at 2026-08-12 14:22:47 on dist-test-slave-8w9v"
server_key: "0e0ae7d746d48ec30fa8c5189d7de5f3"
server_key_iv: "0e5558036569187bb01e402bea3c9aeb"
server_key_version: "encryptionkey@0"
I20260812 14:22:47.198349 9523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:47.199115 9538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:47.199308 9523 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:22:47.199422 9523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "725d070008e943c6936a7fca45520879"
format_stamp: "Formatted at 2026-08-12 14:22:47 on dist-test-slave-8w9v"
server_key: "0e0ae7d746d48ec30fa8c5189d7de5f3"
server_key_iv: "0e5558036569187bb01e402bea3c9aeb"
server_key_version: "encryptionkey@0"
I20260812 14:22:47.199505 9523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:47.219739 9523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:47.222040 9523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:47.222237 9523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:47.222529 9523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:47.222887 9523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:47.222952 9523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:47.222996 9523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:47.223042 9523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:47.232000 9523 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:32803
I20260812 14:22:47.232055 9672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:32803 every 8 connection(s)
I20260812 14:22:47.232599 9523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:22:47.237150 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 9523
I20260812 14:22:47.237277 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:22:47.237490 5712 external_mini_cluster.cc:1428] Setting key 2420cdfd6cfea4e92582ef32b757cfd9
Aug 12 14:22:47 dist-test-slave-8w9v krb5kdc[9239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544567, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:47.244364 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:47.233461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:56535 (local address 127.5.148.62:38625)
0812 14:22:47.233535 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:47.233537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:47.233764 (+ 227us) server_negotiation.cc:407] Connection header received
0812 14:22:47.234342 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:47.234343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:47.234358 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:47.234413 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:47.235577 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:47.236189 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:47.237694 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:47.238112 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:47.238192 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:47.241209 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:47.241211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:47.241212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:47.241220 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:47.242755 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:47.243249 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:47.243251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:47.243251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:47.243318 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:47.243612 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:47.243613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:47.243613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:47.243743 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:47.243824 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:47.244160 (+ 336us) server_negotiation.cc:299] Negotiation successful
0812 14:22:47.244264 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 14:22:47.244607 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:47.233420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38625 (local address 127.5.148.2:56535)
0812 14:22:47.233632 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:47.233675 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:47.234224 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:47.234545 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:47.234549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:47.235230 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:47.235456 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:47.235462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:47.236304 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:47.236305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:47.237429 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:47.237432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:47.238208 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:47.238209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:47.238393 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:47.239008 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:47.239021 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:47.241084 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:47.242873 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:47.242884 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:47.242887 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:47.243163 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:47.243395 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:47.243396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:47.243397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:47.243533 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:47.243814 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:47.243823 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:47.243991 (+ 168us) client_negotiation.cc:769] Sending connection context
0812 14:22:47.244104 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 14:22:47.244323 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":106,"threads_started":1}
I20260812 14:22:47.245270 9673 heartbeater.cc:347] Connected to a master server at 127.5.148.62:38625
I20260812 14:22:47.245420 9673 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:47.245669 9673 heartbeater.cc:511] Master 127.5.148.62:38625 requested a full tablet report, sending...
I20260812 14:22:47.246197 9297 ts_manager.cc:194] Registered new tserver with Master: 725d070008e943c6936a7fca45520879 (127.5.148.2:32803)
I20260812 14:22:47.246778 9297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:56535
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:22:47.267542 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:38625
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:47.361292 9678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:47.361526 9678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:47.361552 9678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:47.362727 9678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:47.362767 9678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:47.362793 9678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:22:47.364573 9678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:38625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:47.364849 9678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:47.365224 9678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:47.370875 9686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:47.370800 9684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:47.370790 9683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:47.371068 9678 server_base.cc:1034] running on GCE node
I20260812 14:22:47.371277 9678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:47.371578 9678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:47.372769 9678 hybrid_clock.cc:648] HybridClock initialized: now 1786544567372755 us; error 31 us; skew 500 ppm
Aug 12 14:22:47 dist-test-slave-8w9v krb5kdc[9239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544567, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:47.375690 9678 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:22:47.376310 9678 webserver.cc:460] Webserver started at http://127.5.148.3:33077/ using document root <none> and password file <none>
I20260812 14:22:47.376590 9678 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:47.376662 9678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:47.376834 9678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:47.377979 9678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "215058b8cbf04ac4a7ae3f395f461091"
format_stamp: "Formatted at 2026-08-12 14:22:47 on dist-test-slave-8w9v"
server_key: "739339b0a0901a480bc81b65615a9e7c"
server_key_iv: "5edb9fcc72b221b59297846436336ff2"
server_key_version: "encryptionkey@0"
I20260812 14:22:47.378377 9678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "215058b8cbf04ac4a7ae3f395f461091"
format_stamp: "Formatted at 2026-08-12 14:22:47 on dist-test-slave-8w9v"
server_key: "739339b0a0901a480bc81b65615a9e7c"
server_key_iv: "5edb9fcc72b221b59297846436336ff2"
server_key_version: "encryptionkey@0"
I20260812 14:22:47.379866 9678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:22:47.380759 9694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:47.380944 9678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:47.381042 9678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "215058b8cbf04ac4a7ae3f395f461091"
format_stamp: "Formatted at 2026-08-12 14:22:47 on dist-test-slave-8w9v"
server_key: "739339b0a0901a480bc81b65615a9e7c"
server_key_iv: "5edb9fcc72b221b59297846436336ff2"
server_key_version: "encryptionkey@0"
I20260812 14:22:47.381134 9678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:47.402422 9678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:47.404982 9678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:47.405169 9678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:47.405467 9678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:47.405817 9678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:47.405874 9678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:47.405921 9678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:47.405946 9678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:47.414909 9678 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:38637
I20260812 14:22:47.415113 9835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:38637 every 8 connection(s)
I20260812 14:22:47.415455 9678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:22:47 dist-test-slave-8w9v krb5kdc[9239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544567, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:47.424456 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 9678
I20260812 14:22:47.424546 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:22:47.424746 5712 external_mini_cluster.cc:1428] Setting key 59b9139a8aba306221e2314f4b70b456
I20260812 14:22:47.425652 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:47.416279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:57465 (local address 127.5.148.62:38625)
0812 14:22:47.416328 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:47.416329 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:47.416571 (+ 242us) server_negotiation.cc:407] Connection header received
0812 14:22:47.417151 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:47.417152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:47.417167 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:47.417218 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:47.418292 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:47.418817 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:47.420007 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:47.420302 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:47.420396 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:47.422690 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:47.422692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:47.422692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:47.422700 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:47.423981 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:47.424457 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:47.424458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:47.424459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:47.424548 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:47.424854 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:47.424855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:47.424856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:47.425009 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:47.425101 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:47.425458 (+ 357us) server_negotiation.cc:299] Negotiation successful
0812 14:22:47.425574 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:22:47.425904 9838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:47.416241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38625 (local address 127.5.148.3:57465)
0812 14:22:47.416414 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:47.416480 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:47.417037 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:47.417307 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:47.417312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:47.417949 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:47.418178 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:47.418184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:47.418942 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:47.418948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:47.419915 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:47.419917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:47.420415 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:47.420416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:47.420548 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:47.421119 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:47.421129 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:47.422581 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:47.424072 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:47.424082 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:47.424084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:47.424355 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:47.424630 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:47.424631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:47.424631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:47.424753 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:47.425072 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:47.425081 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:47.425295 (+ 214us) client_negotiation.cc:769] Sending connection context
0812 14:22:47.425400 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 14:22:47.425622 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260812 14:22:47.426491 9836 heartbeater.cc:347] Connected to a master server at 127.5.148.62:38625
I20260812 14:22:47.426586 9836 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:47.426743 9836 heartbeater.cc:511] Master 127.5.148.62:38625 requested a full tablet report, sending...
I20260812 14:22:47.427093 9297 ts_manager.cc:194] Registered new tserver with Master: 215058b8cbf04ac4a7ae3f395f461091 (127.5.148.3:38637)
I20260812 14:22:47.427510 9297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:57465
I20260812 14:22:47.437243 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 14:22:47 dist-test-slave-8w9v krb5kdc[9239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544567, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 14:22:47 dist-test-slave-8w9v krb5kdc[9239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544567, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:47.463460 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:47.455873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47052 (local address 127.5.148.62:38625)
0812 14:22:47.455958 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:47.455960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:47.456083 (+ 123us) server_negotiation.cc:407] Connection header received
0812 14:22:47.456210 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:47.456211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:47.456222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:47.456269 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:47.456880 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:47.457396 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:47.458153 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:47.458339 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:47.458391 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:47.460769 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:47.460771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:47.460772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:47.460779 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:47.462433 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:47.462758 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:47.462759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:47.462759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:47.462815 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:47.462977 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:47.462978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:47.462978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:47.463137 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:47.463219 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:47.463312 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 14:22:47.463377 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 14:22:47.463732 9297 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47052
I20260812 14:22:47.464339 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 9369
I20260812 14:22:47.474540 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 9523
I20260812 14:22:47.481736 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 9678
I20260812 14:22:47.488950 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 9257
2026-08-12T14:22:47Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2102 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:22:47 dist-test-slave-8w9v krb5kdc[9850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:22:47 dist-test-slave-8w9v krb5kdc[9850](info): set up 2 sockets
Aug 12 14:22:47 dist-test-slave-8w9v krb5kdc[9850](info): commencing operation
krb5kdc: starting...
W20260812 14:22:48.620959 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:22:48 dist-test-slave-8w9v krb5kdc[9850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:22:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:22:48.755596 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:41243
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:48.841677 9866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:48.841926 9866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:48.841966 9866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:48.843122 9866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:48.843195 9866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:48.843219 9866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:48.843251 9866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:48.843281 9866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:22:48.844892 9866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:41243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:41243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:48.845189 9866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:48.845438 9866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:48.850667 9871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:48.850715 9872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:48.850648 9875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:48.850910 9866 server_base.cc:1034] running on GCE node
I20260812 14:22:48.851186 9866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:48.851532 9866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:48.852707 9866 hybrid_clock.cc:648] HybridClock initialized: now 1786544568852687 us; error 51 us; skew 500 ppm
Aug 12 14:22:48 dist-test-slave-8w9v krb5kdc[9850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544568, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:48.862736 9866 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:22:48.863337 9866 webserver.cc:460] Webserver started at http://127.5.148.62:32885/ using document root <none> and password file <none>
I20260812 14:22:48.863613 9866 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:48.863684 9866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:48.863849 9866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:48.865073 9866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "a9f98d5da4d24bc5ae082cb009ea4a81"
format_stamp: "Formatted at 2026-08-12 14:22:48 on dist-test-slave-8w9v"
server_key: "cf802e3d34c9c2304fcb7751ea160919"
server_key_iv: "82b27ac0f35105351b1c687676ef992e"
server_key_version: "encryptionkey@0"
I20260812 14:22:48.865460 9866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "a9f98d5da4d24bc5ae082cb009ea4a81"
format_stamp: "Formatted at 2026-08-12 14:22:48 on dist-test-slave-8w9v"
server_key: "cf802e3d34c9c2304fcb7751ea160919"
server_key_iv: "82b27ac0f35105351b1c687676ef992e"
server_key_version: "encryptionkey@0"
I20260812 14:22:48.866931 9866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:22:48.867695 9882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:48.867902 9866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:48.868019 9866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "a9f98d5da4d24bc5ae082cb009ea4a81"
format_stamp: "Formatted at 2026-08-12 14:22:48 on dist-test-slave-8w9v"
server_key: "cf802e3d34c9c2304fcb7751ea160919"
server_key_iv: "82b27ac0f35105351b1c687676ef992e"
server_key_version: "encryptionkey@0"
I20260812 14:22:48.868098 9866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:48.880103 9866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:48.882710 9866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:48.882896 9866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:48.887197 9866 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:41243
I20260812 14:22:48.887243 9940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:41243 every 8 connection(s)
I20260812 14:22:48.887847 9866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:48.888623 9941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:48.891000 9941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81: Bootstrap starting.
I20260812 14:22:48.891690 9941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:48.891742 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 9866
I20260812 14:22:48.891862 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:48.892083 5712 external_mini_cluster.cc:1428] Setting key e5aa04171ee3e81a65e15d7bc03c2333
I20260812 14:22:48.892063 9941 log.cc:824] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:48.893005 9941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81: No bootstrap required, opened a new log
I20260812 14:22:48.895778 9941 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f98d5da4d24bc5ae082cb009ea4a81" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 41243 } }
I20260812 14:22:48.895959 9941 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:48.896011 9941 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f98d5da4d24bc5ae082cb009ea4a81, State: Initialized, Role: FOLLOWER
I20260812 14:22:48.896163 9941 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f98d5da4d24bc5ae082cb009ea4a81" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 41243 } }
I20260812 14:22:48.896242 9941 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:48.896270 9941 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:48.896309 9941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:48.896896 9941 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f98d5da4d24bc5ae082cb009ea4a81" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 41243 } }
I20260812 14:22:48.897039 9941 leader_election.cc:302] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9f98d5da4d24bc5ae082cb009ea4a81; no voters:
I20260812 14:22:48.897217 9941 leader_election.cc:288] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:48.897289 9948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:48.897471 9948 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [term 1 LEADER]: Becoming Leader. State: Replica: a9f98d5da4d24bc5ae082cb009ea4a81, State: Running, Role: LEADER
I20260812 14:22:48.897612 9941 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:48.897651 9948 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f98d5da4d24bc5ae082cb009ea4a81" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 41243 } }
I20260812 14:22:48.898038 9949 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9f98d5da4d24bc5ae082cb009ea4a81. Latest consensus state: current_term: 1 leader_uuid: "a9f98d5da4d24bc5ae082cb009ea4a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f98d5da4d24bc5ae082cb009ea4a81" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 41243 } } }
I20260812 14:22:48.898187 9949 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:48.898658 9956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:22:48.898852 9950 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9f98d5da4d24bc5ae082cb009ea4a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f98d5da4d24bc5ae082cb009ea4a81" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 41243 } } }
I20260812 14:22:48.899032 9950 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:48.899160 9956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 14:22:48 dist-test-slave-8w9v krb5kdc[9850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:48.900902 9956 catalog_manager.cc:1347] Generated new cluster ID: 137d6d45966a45f8a35d5dbf1647b4ca
I20260812 14:22:48.900959 9956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:22:48.904323 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:48.892730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54180 (local address 127.5.148.62:41243)
0812 14:22:48.892916 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:48.892926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:48.892959 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:22:48.893504 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:48.893512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:48.893562 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:48.893678 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:48.894513 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:48.895365 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:48.896489 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:48.896902 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:48.896982 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:48.900053 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:48.900058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:48.900059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:48.900071 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:48.902189 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:48.902618 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:48.902621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:48.902622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:48.902727 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:48.902962 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:48.902963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:48.902963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:48.903276 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:48.903441 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:48.903918 (+ 477us) server_negotiation.cc:299] Negotiation successful
0812 14:22:48.904061 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":118,"threads_started":1}
I20260812 14:22:48.920154 9956 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:48.920948 9956 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:48.932703 9956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9f98d5da4d24bc5ae082cb009ea4a81: Generated new TSK 0
I20260812 14:22:48.932966 9956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:22:48.965528 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:41243
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:49.051311 9975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:49.051560 9975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:49.051601 9975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:49.052805 9975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:49.052877 9975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:49.052922 9975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:22:49.054466 9975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:49.054737 9975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:49.054986 9975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:49.060122 9984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:49.060284 9975 server_base.cc:1034] running on GCE node
W20260812 14:22:49.060093 9982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:49.060088 9981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:49.060766 9975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:49.061041 9975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:49.062211 9975 hybrid_clock.cc:648] HybridClock initialized: now 1786544569062204 us; error 35 us; skew 500 ppm
Aug 12 14:22:49 dist-test-slave-8w9v krb5kdc[9850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544569, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:49.064960 9975 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:22:49.065429 9975 webserver.cc:460] Webserver started at http://127.5.148.1:41493/ using document root <none> and password file <none>
I20260812 14:22:49.065686 9975 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:49.065757 9975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:49.065917 9975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:49.067071 9975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "3dace4c595454323a97b890e58901fa1"
format_stamp: "Formatted at 2026-08-12 14:22:49 on dist-test-slave-8w9v"
server_key: "02753457483feeacaccd304d66579c5f"
server_key_iv: "bc8bf4cc0fe431eba9f95853c0f9b1d3"
server_key_version: "encryptionkey@0"
I20260812 14:22:49.067449 9975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "3dace4c595454323a97b890e58901fa1"
format_stamp: "Formatted at 2026-08-12 14:22:49 on dist-test-slave-8w9v"
server_key: "02753457483feeacaccd304d66579c5f"
server_key_iv: "bc8bf4cc0fe431eba9f95853c0f9b1d3"
server_key_version: "encryptionkey@0"
I20260812 14:22:49.068889 9975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:22:49.069609 9994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:49.069789 9975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:22:49.069898 9975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "3dace4c595454323a97b890e58901fa1"
format_stamp: "Formatted at 2026-08-12 14:22:49 on dist-test-slave-8w9v"
server_key: "02753457483feeacaccd304d66579c5f"
server_key_iv: "bc8bf4cc0fe431eba9f95853c0f9b1d3"
server_key_version: "encryptionkey@0"
I20260812 14:22:49.069977 9975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:49.103679 9975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:49.105998 9975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:49.106181 9975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:49.106488 9975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:49.106832 9975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:49.106887 9975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:49.106931 9975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:49.106977 9975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:49.114012 9975 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:35945
I20260812 14:22:49.114099 10129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:35945 every 8 connection(s)
I20260812 14:22:49.114495 9975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:22:49 dist-test-slave-8w9v krb5kdc[9850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544569, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:49.122471 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 9975
I20260812 14:22:49.122568 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:22:49.122763 5712 external_mini_cluster.cc:1428] Setting key 285f1e7d6215c48686e71a674c7db675
I20260812 14:22:49.125345 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:49.115418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:36273 (local address 127.5.148.62:41243)
0812 14:22:49.115526 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:49.115528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:49.115833 (+ 305us) server_negotiation.cc:407] Connection header received
0812 14:22:49.116969 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:49.116971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:49.116984 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:49.117044 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:49.117788 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.118319 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.119363 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.119593 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.119674 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:49.121872 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:49.121873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:49.121874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:49.121881 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:49.123696 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:49.124236 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:49.124237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:49.124237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:49.124309 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:49.124600 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:49.124601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:49.124601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:49.124733 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:49.124827 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:49.125134 (+ 307us) server_negotiation.cc:299] Negotiation successful
0812 14:22:49.125241 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 14:22:49.125553 10132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:49.115374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:41243 (local address 127.5.148.1:36273)
0812 14:22:49.115670 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:49.115723 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:49.116842 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:49.117115 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:49.117119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:49.117424 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:49.117658 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:49.117663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.118423 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.118424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:49.119242 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:49.119244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.119683 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.119684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:49.119825 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:49.120392 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:49.120400 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:49.121770 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:49.123815 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:49.123824 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:49.123826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:49.124107 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:49.124393 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:49.124394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:49.124394 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:49.124516 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:49.124817 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:49.124825 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:49.124985 (+ 160us) client_negotiation.cc:769] Sending connection context
0812 14:22:49.125098 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 14:22:49.125285 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":152,"threads_started":1}
I20260812 14:22:49.126276 10130 heartbeater.cc:347] Connected to a master server at 127.5.148.62:41243
I20260812 14:22:49.126398 10130 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:49.126638 10130 heartbeater.cc:511] Master 127.5.148.62:41243 requested a full tablet report, sending...
I20260812 14:22:49.127230 9901 ts_manager.cc:194] Registered new tserver with Master: 3dace4c595454323a97b890e58901fa1 (127.5.148.1:35945)
I20260812 14:22:49.127857 9901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:36273
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:22:49.153589 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:41243
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:49.246671 10136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:49.246950 10136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:49.246992 10136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:49.248323 10136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:49.248399 10136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:49.248445 10136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:22:49.249989 10136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:49.250290 10136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:49.250543 10136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:49.255927 10144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:49.255927 10147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:49.255976 10145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:49.256008 10136 server_base.cc:1034] running on GCE node
I20260812 14:22:49.256512 10136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:49.256821 10136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:49.257980 10136 hybrid_clock.cc:648] HybridClock initialized: now 1786544569257970 us; error 30 us; skew 500 ppm
Aug 12 14:22:49 dist-test-slave-8w9v krb5kdc[9850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544569, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:49.260872 10136 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:22:49.261386 10136 webserver.cc:460] Webserver started at http://127.5.148.2:40839/ using document root <none> and password file <none>
I20260812 14:22:49.261623 10136 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:49.261693 10136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:49.261850 10136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:49.262982 10136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "5d52dd64f3db4bc18d0c4321bbac1bc3"
format_stamp: "Formatted at 2026-08-12 14:22:49 on dist-test-slave-8w9v"
server_key: "f2272ce00c4b51a8251a34b7377fcd8e"
server_key_iv: "46a28fa369063a01c53092e24ad6186c"
server_key_version: "encryptionkey@0"
I20260812 14:22:49.263381 10136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "5d52dd64f3db4bc18d0c4321bbac1bc3"
format_stamp: "Formatted at 2026-08-12 14:22:49 on dist-test-slave-8w9v"
server_key: "f2272ce00c4b51a8251a34b7377fcd8e"
server_key_iv: "46a28fa369063a01c53092e24ad6186c"
server_key_version: "encryptionkey@0"
I20260812 14:22:49.264922 10136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:22:49.265731 10155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:49.265903 10136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:22:49.266017 10136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "5d52dd64f3db4bc18d0c4321bbac1bc3"
format_stamp: "Formatted at 2026-08-12 14:22:49 on dist-test-slave-8w9v"
server_key: "f2272ce00c4b51a8251a34b7377fcd8e"
server_key_iv: "46a28fa369063a01c53092e24ad6186c"
server_key_version: "encryptionkey@0"
I20260812 14:22:49.266098 10136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:49.287335 10136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:49.289885 10136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:49.290120 10136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:49.290518 10136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:49.290872 10136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:49.290936 10136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:49.290982 10136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:49.291010 10136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:49.298256 10136 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:41427
I20260812 14:22:49.298408 10290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:41427 every 8 connection(s)
I20260812 14:22:49.298722 10136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:22:49.300197 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 10136
I20260812 14:22:49.300321 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:22:49.300519 5712 external_mini_cluster.cc:1428] Setting key d80d06ca26617b820f301e9d1d55e7a4
Aug 12 14:22:49 dist-test-slave-8w9v krb5kdc[9850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544569, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:49.309765 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:49.299532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:45383 (local address 127.5.148.62:41243)
0812 14:22:49.299603 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:49.299604 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:49.299812 (+ 208us) server_negotiation.cc:407] Connection header received
0812 14:22:49.300896 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:49.300897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:49.300909 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:49.300965 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:49.301666 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.302432 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.303604 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.303818 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.303914 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:49.306540 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:49.306542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:49.306542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:49.306550 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:49.308308 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:49.308735 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:49.308736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:49.308737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:49.308796 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:49.309084 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:49.309085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:49.309085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:49.309212 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:49.309284 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:49.309581 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 14:22:49.309669 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 14:22:49.309968 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:49.299505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:41243 (local address 127.5.148.2:45383)
0812 14:22:49.299668 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:49.299714 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:49.300810 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:49.301044 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:49.301049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:49.301368 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:49.301569 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:49.301574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.302562 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.302563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:49.303477 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:49.303480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.303929 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.303930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:49.304062 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:49.304703 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:49.304712 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:49.306429 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:49.308406 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:49.308413 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:49.308416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:49.308655 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:49.308889 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:49.308890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:49.308890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:49.309007 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:49.309287 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:49.309295 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:49.309447 (+ 152us) client_negotiation.cc:769] Sending connection context
0812 14:22:49.309559 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 14:22:49.309725 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
I20260812 14:22:49.310451 10291 heartbeater.cc:347] Connected to a master server at 127.5.148.62:41243
I20260812 14:22:49.310585 10291 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:49.310770 10291 heartbeater.cc:511] Master 127.5.148.62:41243 requested a full tablet report, sending...
I20260812 14:22:49.311182 9901 ts_manager.cc:194] Registered new tserver with Master: 5d52dd64f3db4bc18d0c4321bbac1bc3 (127.5.148.2:41427)
I20260812 14:22:49.311609 9901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:45383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:22:49.331449 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:41243
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:49.422797 10297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:49.423055 10297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:49.423099 10297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:49.424404 10297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:49.424494 10297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:49.424542 10297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:22:49.426258 10297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:49.426586 10297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:49.426857 10297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:49.432711 10303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:49.432947 10297 server_base.cc:1034] running on GCE node
W20260812 14:22:49.432739 10306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:49.432837 10302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:49.433321 10297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:49.433636 10297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:49.434839 10297 hybrid_clock.cc:648] HybridClock initialized: now 1786544569434816 us; error 37 us; skew 500 ppm
Aug 12 14:22:49 dist-test-slave-8w9v krb5kdc[9850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544569, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:49.438061 10297 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:22:49.438563 10297 webserver.cc:460] Webserver started at http://127.5.148.3:41787/ using document root <none> and password file <none>
I20260812 14:22:49.438831 10297 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:49.438901 10297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:49.439055 10297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:49.440258 10297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "21ff98348b774681b96e0acaf1e7f463"
format_stamp: "Formatted at 2026-08-12 14:22:49 on dist-test-slave-8w9v"
server_key: "1191b7a8efe85d1c61f4ea0b2f88528a"
server_key_iv: "591f4644cdd1445225e1da3237b4fcfc"
server_key_version: "encryptionkey@0"
I20260812 14:22:49.440646 10297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "21ff98348b774681b96e0acaf1e7f463"
format_stamp: "Formatted at 2026-08-12 14:22:49 on dist-test-slave-8w9v"
server_key: "1191b7a8efe85d1c61f4ea0b2f88528a"
server_key_iv: "591f4644cdd1445225e1da3237b4fcfc"
server_key_version: "encryptionkey@0"
I20260812 14:22:49.442127 10297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:22:49.442987 10315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:49.443167 10297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:49.443282 10297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "21ff98348b774681b96e0acaf1e7f463"
format_stamp: "Formatted at 2026-08-12 14:22:49 on dist-test-slave-8w9v"
server_key: "1191b7a8efe85d1c61f4ea0b2f88528a"
server_key_iv: "591f4644cdd1445225e1da3237b4fcfc"
server_key_version: "encryptionkey@0"
I20260812 14:22:49.443394 10297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:49.460512 10297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:49.463035 10297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:49.463228 10297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:49.463508 10297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:49.463939 10297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:49.464000 10297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:49.464037 10297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:49.464079 10297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:49.472157 10297 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:40015
I20260812 14:22:49.472205 10450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:40015 every 8 connection(s)
I20260812 14:22:49.472653 10297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:22:49.477941 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 10297
I20260812 14:22:49.478091 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:22:49.478434 5712 external_mini_cluster.cc:1428] Setting key 3bbb9d82c5c277364bdec02105a278a0
Aug 12 14:22:49 dist-test-slave-8w9v krb5kdc[9850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544569, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:49.483567 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:49.473486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:54301 (local address 127.5.148.62:41243)
0812 14:22:49.473573 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:49.473575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:49.473840 (+ 265us) server_negotiation.cc:407] Connection header received
0812 14:22:49.474972 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:49.474973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:49.474986 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:49.475079 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:49.475783 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.476314 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.477780 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.477971 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.478027 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:49.481062 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:49.481064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:49.481064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:49.481072 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:49.482318 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:49.482745 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:49.482746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:49.482746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:49.482800 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:49.483024 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:49.483024 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:49.483024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:49.483131 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:49.483182 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:49.483429 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 14:22:49.483499 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 14:22:49.483829 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:49.473444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:41243 (local address 127.5.148.3:54301)
0812 14:22:49.473656 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:49.473709 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:49.474795 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:49.475157 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:49.475160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:49.475489 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:49.475700 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:49.475704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.476443 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.476444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:49.477590 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:49.477593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.478084 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.478086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:49.478321 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:49.479150 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:49.479165 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:49.480943 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:49.482440 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:49.482447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:49.482449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:49.482676 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:49.482860 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:49.482861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:49.482861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:49.482967 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:49.483187 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:49.483192 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:49.483316 (+ 124us) client_negotiation.cc:769] Sending connection context
0812 14:22:49.483400 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 14:22:49.483543 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260812 14:22:49.484459 10451 heartbeater.cc:347] Connected to a master server at 127.5.148.62:41243
I20260812 14:22:49.484583 10451 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:49.484781 10451 heartbeater.cc:511] Master 127.5.148.62:41243 requested a full tablet report, sending...
I20260812 14:22:49.485208 9901 ts_manager.cc:194] Registered new tserver with Master: 21ff98348b774681b96e0acaf1e7f463 (127.5.148.3:40015)
I20260812 14:22:49.485613 9901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:54301
I20260812 14:22:49.490634 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 14:22:49 dist-test-slave-8w9v krb5kdc[9850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 14:22:49 dist-test-slave-8w9v krb5kdc[9850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.1@KRBTEST.COM
I20260812 14:22:49.516330 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:49.508420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.5.148.1:35945)
0812 14:22:49.508608 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:49.508614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:49.508626 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:22:49.508657 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:49.508659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:49.508673 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:49.508742 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:49.509398 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.510139 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.510869 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.511216 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.511271 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:49.513636 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:49.513646 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:49.513647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:49.513670 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:49.515197 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:49.515523 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:49.515527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:49.515527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:49.515586 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:49.515753 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:49.515753 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:49.515754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:49.515890 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:49.516031 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:49.516073 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 14:22:49.516209 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
Aug 12 14:22:49 dist-test-slave-8w9v krb5kdc[9850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:49.524704 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:49.517722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.5.148.62:41243)
0812 14:22:49.517782 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:49.517784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:49.517794 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:22:49.517825 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:49.517826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:49.517834 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:49.517882 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:49.518367 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.518852 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.519503 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:49.519681 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:49.519732 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:49.522059 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:49.522060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:49.522061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:49.522068 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:49.523537 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:49.523863 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:49.523864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:49.523864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:49.523916 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:49.524095 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:49.524096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:49.524096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:49.524380 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:49.524464 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:49.524569 (+ 105us) server_negotiation.cc:299] Negotiation successful
0812 14:22:49.524634 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260812 14:22:49.524966 9901 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54182
I20260812 14:22:49.525362 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 9975
I20260812 14:22:49.534126 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 10136
I20260812 14:22:49.545085 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 10297
I20260812 14:22:49.554458 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 9866
2026-08-12T14:22:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2063 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:22:49 dist-test-slave-8w9v krb5kdc[10464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:22:49 dist-test-slave-8w9v krb5kdc[10464](info): set up 2 sockets
Aug 12 14:22:49 dist-test-slave-8w9v krb5kdc[10464](info): commencing operation
krb5kdc: starting...
W20260812 14:22:50.688735 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:22:50 dist-test-slave-8w9v krb5kdc[10464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:22:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:22:50.821485 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:46653
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:35005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:46653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:50.908320 10481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:50.908545 10481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:50.908569 10481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:50.909775 10481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:50.909823 10481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:50.909838 10481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:50.909852 10481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:50.909905 10481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:22:50.911481 10481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:35005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:46653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:46653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:50.911746 10481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:50.911985 10481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:50.917007 10486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:50.916996 10489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:50.916981 10487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:50.917354 10481 server_base.cc:1034] running on GCE node
I20260812 14:22:50.917650 10481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:50.917976 10481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:50.919150 10481 hybrid_clock.cc:648] HybridClock initialized: now 1786544570919137 us; error 27 us; skew 500 ppm
Aug 12 14:22:50 dist-test-slave-8w9v krb5kdc[10464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544570, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:50.921792 10481 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:22:50.922155 10481 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:50.922231 10481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:50.922384 10481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:50.923524 10481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "8e854119cf4e4821afba3c9a65db80d2"
format_stamp: "Formatted at 2026-08-12 14:22:50 on dist-test-slave-8w9v"
server_key: "c1407676826b4a7245338d3987d32bdf"
server_key_iv: "17529b94af18d5e36e3d6a996c001a19"
server_key_version: "encryptionkey@0"
I20260812 14:22:50.923915 10481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "8e854119cf4e4821afba3c9a65db80d2"
format_stamp: "Formatted at 2026-08-12 14:22:50 on dist-test-slave-8w9v"
server_key: "c1407676826b4a7245338d3987d32bdf"
server_key_iv: "17529b94af18d5e36e3d6a996c001a19"
server_key_version: "encryptionkey@0"
I20260812 14:22:50.925335 10481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:50.926115 10496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:50.926298 10481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:22:50.926429 10481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "8e854119cf4e4821afba3c9a65db80d2"
format_stamp: "Formatted at 2026-08-12 14:22:50 on dist-test-slave-8w9v"
server_key: "c1407676826b4a7245338d3987d32bdf"
server_key_iv: "17529b94af18d5e36e3d6a996c001a19"
server_key_version: "encryptionkey@0"
I20260812 14:22:50.926512 10481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:50.944432 10481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:50.946558 10481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:50.946744 10481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:50.951051 10481 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:46653
I20260812 14:22:50.951084 10559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:46653 every 8 connection(s)
I20260812 14:22:50.951406 10481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:50.952436 10560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:50.954937 10560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2: Bootstrap starting.
I20260812 14:22:50.955588 10560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:50.955910 10560 log.cc:824] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:50.956738 10560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2: No bootstrap required, opened a new log
I20260812 14:22:50.957438 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 10481
I20260812 14:22:50.957571 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:50.957758 5712 external_mini_cluster.cc:1428] Setting key eb6a5c5ca84160586f19a713adf901f5
I20260812 14:22:50.959271 10560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e854119cf4e4821afba3c9a65db80d2" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46653 } }
I20260812 14:22:50.959465 10560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:50.959506 10560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e854119cf4e4821afba3c9a65db80d2, State: Initialized, Role: FOLLOWER
I20260812 14:22:50.959690 10560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e854119cf4e4821afba3c9a65db80d2" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46653 } }
I20260812 14:22:50.959789 10560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:50.959822 10560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:50.959879 10560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:50.960633 10560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e854119cf4e4821afba3c9a65db80d2" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46653 } }
I20260812 14:22:50.960727 10560 leader_election.cc:302] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e854119cf4e4821afba3c9a65db80d2; no voters:
I20260812 14:22:50.960922 10560 leader_election.cc:288] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:50.960963 10567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:50.961079 10567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [term 1 LEADER]: Becoming Leader. State: Replica: 8e854119cf4e4821afba3c9a65db80d2, State: Running, Role: LEADER
I20260812 14:22:50.961223 10560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:50.961232 10567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e854119cf4e4821afba3c9a65db80d2" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46653 } }
I20260812 14:22:50.961640 10567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e854119cf4e4821afba3c9a65db80d2. Latest consensus state: current_term: 1 leader_uuid: "8e854119cf4e4821afba3c9a65db80d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e854119cf4e4821afba3c9a65db80d2" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46653 } } }
I20260812 14:22:50.961649 10570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e854119cf4e4821afba3c9a65db80d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e854119cf4e4821afba3c9a65db80d2" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46653 } } }
I20260812 14:22:50.961810 10567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:50.962062 10570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:50.962745 10577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:22:50.963515 10577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:22:50.965509 10577 catalog_manager.cc:1347] Generated new cluster ID: 1918c3bdbc004d949e7f4640d68ac33a
I20260812 14:22:50.965564 10577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:22:50 dist-test-slave-8w9v krb5kdc[10464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:50.971022 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:50.958365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.5.148.62:46653)
0812 14:22:50.958557 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:50.958564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:50.958591 (+ 27us) server_negotiation.cc:407] Connection header received
0812 14:22:50.959250 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:50.959258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:50.959308 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:50.959441 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:50.960333 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:50.961697 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:50.962909 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:50.963433 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:50.963561 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:50.966522 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:50.966527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:50.966527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:50.966539 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:50.968697 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:50.969130 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:50.969134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:50.969134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:50.969271 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:50.969495 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:50.969496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:50.969496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:50.969835 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:50.970544 (+ 709us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:50.970627 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 14:22:50.970786 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260812 14:22:50.980564 10577 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:50.981179 10577 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:50.984557 10577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2: Generated new TSK 0
I20260812 14:22:50.984778 10577 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:22:51.022984 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:46653
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:35005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:51.126386 10591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:51.126595 10591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:51.126619 10591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:51.127784 10591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:51.127827 10591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:51.127861 10591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:22:51.129698 10591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:35005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:46653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:51.129947 10591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:51.130146 10591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:51.135236 10596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:51.135327 10600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:51.135252 10598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:51.135458 10591 server_base.cc:1034] running on GCE node
I20260812 14:22:51.135706 10591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:51.135980 10591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:51.137163 10591 hybrid_clock.cc:648] HybridClock initialized: now 1786544571137147 us; error 45 us; skew 500 ppm
Aug 12 14:22:51 dist-test-slave-8w9v krb5kdc[10464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544571, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:51.139927 10591 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:22:51.140345 10591 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:51.140401 10591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:51.140496 10591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:51.141530 10591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "f7141f1c555a4224b38b80eb81d61cbc"
format_stamp: "Formatted at 2026-08-12 14:22:51 on dist-test-slave-8w9v"
server_key: "f985de9a39d0701a0c1ea697f6c98dac"
server_key_iv: "1eb36eeaee734ed4e20500e0f05fbfd4"
server_key_version: "encryptionkey@0"
I20260812 14:22:51.141835 10591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "f7141f1c555a4224b38b80eb81d61cbc"
format_stamp: "Formatted at 2026-08-12 14:22:51 on dist-test-slave-8w9v"
server_key: "f985de9a39d0701a0c1ea697f6c98dac"
server_key_iv: "1eb36eeaee734ed4e20500e0f05fbfd4"
server_key_version: "encryptionkey@0"
I20260812 14:22:51.143088 10591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:22:51.143790 10606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:51.143988 10591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:51.144076 10591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "f7141f1c555a4224b38b80eb81d61cbc"
format_stamp: "Formatted at 2026-08-12 14:22:51 on dist-test-slave-8w9v"
server_key: "f985de9a39d0701a0c1ea697f6c98dac"
server_key_iv: "1eb36eeaee734ed4e20500e0f05fbfd4"
server_key_version: "encryptionkey@0"
I20260812 14:22:51.144194 10591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:51.157325 10591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:51.159304 10591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:51.159425 10591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:51.159628 10591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:51.159973 10591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:51.160009 10591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:51.160033 10591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:51.160077 10591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:51.167900 10591 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:43241
I20260812 14:22:51.167982 10740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:43241 every 8 connection(s)
I20260812 14:22:51.168300 10591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:22:51.170372 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 10591
I20260812 14:22:51.170459 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:22:51.170657 5712 external_mini_cluster.cc:1428] Setting key d3aff4b013fa5a3026348cbddce3a786
Aug 12 14:22:51 dist-test-slave-8w9v krb5kdc[10464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544571, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:51.179592 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.169075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:42005 (local address 127.5.148.62:46653)
0812 14:22:51.169161 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:51.169164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:51.169361 (+ 197us) server_negotiation.cc:407] Connection header received
0812 14:22:51.169858 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:51.169859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:51.169874 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:51.169928 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:51.171133 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.171719 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.173337 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.173695 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.173771 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.176049 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:51.176051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:51.176051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:51.176063 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:51.177774 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:51.178363 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:51.178364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:51.178365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:51.178435 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:51.178782 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:51.178783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:51.178783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:51.178922 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:51.178998 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:51.179425 (+ 427us) server_negotiation.cc:299] Negotiation successful
0812 14:22:51.179520 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 14:22:51.179818 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.169040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:46653 (local address 127.5.148.1:42005)
0812 14:22:51.169228 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:51.169267 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:51.169767 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:51.169982 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:51.169985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:51.170252 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:51.171031 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.171038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.171820 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.171821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.173204 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.173208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.173795 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.173795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.173947 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.174524 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:51.174532 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:51.175935 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:51.177869 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:51.177877 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:51.177879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:51.178237 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:51.178510 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:51.178511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:51.178512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:51.178670 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:51.179014 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:51.179023 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:51.179210 (+ 187us) client_negotiation.cc:769] Sending connection context
0812 14:22:51.179346 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 14:22:51.179539 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260812 14:22:51.180454 10741 heartbeater.cc:347] Connected to a master server at 127.5.148.62:46653
I20260812 14:22:51.180581 10741 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:51.180765 10741 heartbeater.cc:511] Master 127.5.148.62:46653 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
I20260812 14:22:51.181306 10516 ts_manager.cc:194] Registered new tserver with Master: f7141f1c555a4224b38b80eb81d61cbc (127.5.148.1:43241)
I20260812 14:22:51.181929 10516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:42005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:22:51.199970 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:46653
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:35005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:51.290825 10746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:51.291117 10746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:51.291162 10746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:51.292452 10746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:51.292541 10746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:51.292590 10746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:22:51.294337 10746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:35005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:46653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:51.294646 10746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:51.294927 10746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:51.300577 10753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:51.300621 10755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:51.300678 10752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:51.301239 10746 server_base.cc:1034] running on GCE node
I20260812 14:22:51.301491 10746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:51.301699 10746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:51.302870 10746 hybrid_clock.cc:648] HybridClock initialized: now 1786544571302854 us; error 35 us; skew 500 ppm
Aug 12 14:22:51 dist-test-slave-8w9v krb5kdc[10464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544571, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:51.305680 10746 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:22:51.306066 10746 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:51.306138 10746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:51.306291 10746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:51.307399 10746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "345638ef66c046b58fdb128baa2ad836"
format_stamp: "Formatted at 2026-08-12 14:22:51 on dist-test-slave-8w9v"
server_key: "47eccd3f79c38c7229713f4e41eb208a"
server_key_iv: "68914f12d4bb7c8a64e154f64013bea9"
server_key_version: "encryptionkey@0"
I20260812 14:22:51.307840 10746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "345638ef66c046b58fdb128baa2ad836"
format_stamp: "Formatted at 2026-08-12 14:22:51 on dist-test-slave-8w9v"
server_key: "47eccd3f79c38c7229713f4e41eb208a"
server_key_iv: "68914f12d4bb7c8a64e154f64013bea9"
server_key_version: "encryptionkey@0"
I20260812 14:22:51.309484 10746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:22:51.310354 10761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:51.310551 10746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:22:51.310660 10746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "345638ef66c046b58fdb128baa2ad836"
format_stamp: "Formatted at 2026-08-12 14:22:51 on dist-test-slave-8w9v"
server_key: "47eccd3f79c38c7229713f4e41eb208a"
server_key_iv: "68914f12d4bb7c8a64e154f64013bea9"
server_key_version: "encryptionkey@0"
I20260812 14:22:51.310736 10746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:51.321226 10746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:51.323354 10746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:51.323529 10746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:51.323779 10746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:51.324204 10746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:51.324268 10746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:51.324306 10746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:51.324378 10746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:51.332839 10746 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:45801
I20260812 14:22:51.332880 10895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:45801 every 8 connection(s)
I20260812 14:22:51.333189 10746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:22:51.336601 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 10746
I20260812 14:22:51.336742 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:22:51.336958 5712 external_mini_cluster.cc:1428] Setting key 6dc6e71553e9a658035b15646bc10aa0
Aug 12 14:22:51 dist-test-slave-8w9v krb5kdc[10464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544571, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:51.345659 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.334013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:57871 (local address 127.5.148.62:46653)
0812 14:22:51.334102 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:51.334104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:51.334548 (+ 444us) server_negotiation.cc:407] Connection header received
0812 14:22:51.335219 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:51.335220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:51.335233 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:51.335296 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:51.336829 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.337533 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.339285 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.339510 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.339577 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.342075 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:51.342077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:51.342078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:51.342087 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:51.343937 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:51.344479 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:51.344481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:51.344481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:51.344553 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:51.344824 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:51.344826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:51.344826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:51.345011 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:51.345087 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:51.345428 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 14:22:51.345540 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 14:22:51.345695 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.333969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:46653 (local address 127.5.148.2:57871)
0812 14:22:51.334262 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:51.334304 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:51.335074 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:51.335357 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:51.335360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:51.335690 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:51.336705 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.336716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.337649 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.337650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.339173 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.339177 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.339607 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.339608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.339771 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.340480 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:51.340490 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:51.341972 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:51.344043 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:51.344056 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:51.344058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:51.344400 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:51.344631 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:51.344631 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:51.344632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:51.344741 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:51.345088 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:51.345095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:51.345228 (+ 133us) client_negotiation.cc:769] Sending connection context
0812 14:22:51.345358 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 14:22:51.345508 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":109,"threads_started":1}
I20260812 14:22:51.346256 10896 heartbeater.cc:347] Connected to a master server at 127.5.148.62:46653
I20260812 14:22:51.346371 10896 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:51.346603 10896 heartbeater.cc:511] Master 127.5.148.62:46653 requested a full tablet report, sending...
I20260812 14:22:51.347074 10516 ts_manager.cc:194] Registered new tserver with Master: 345638ef66c046b58fdb128baa2ad836 (127.5.148.2:45801)
I20260812 14:22:51.347616 10516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:57871
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:22:51.368240 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:46653
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:35005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:51.463369 10901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:51.463640 10901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:51.463687 10901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:51.465040 10901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:51.465121 10901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:51.465173 10901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:22:51.466845 10901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:35005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:46653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:51.467139 10901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:51.467414 10901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:51.473150 10912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:51.473147 10910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:51.473171 10908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:51.473614 10901 server_base.cc:1034] running on GCE node
I20260812 14:22:51.473860 10901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:51.474164 10901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:51.475349 10901 hybrid_clock.cc:648] HybridClock initialized: now 1786544571475344 us; error 38 us; skew 500 ppm
Aug 12 14:22:51 dist-test-slave-8w9v krb5kdc[10464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544571, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:51.478148 10901 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:22:51.478614 10901 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:51.478693 10901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:51.478878 10901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:51.480238 10901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "5794f72b7716440eaaf5321e22bfdf45"
format_stamp: "Formatted at 2026-08-12 14:22:51 on dist-test-slave-8w9v"
server_key: "98e738d0be8d93a1cfe96c0c543b7ac1"
server_key_iv: "8ff3a6142e47ade223b86fcdbf7213d7"
server_key_version: "encryptionkey@0"
I20260812 14:22:51.480675 10901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "5794f72b7716440eaaf5321e22bfdf45"
format_stamp: "Formatted at 2026-08-12 14:22:51 on dist-test-slave-8w9v"
server_key: "98e738d0be8d93a1cfe96c0c543b7ac1"
server_key_iv: "8ff3a6142e47ade223b86fcdbf7213d7"
server_key_version: "encryptionkey@0"
I20260812 14:22:51.482333 10901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:22:51.483335 10918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:51.483548 10901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:51.483651 10901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "5794f72b7716440eaaf5321e22bfdf45"
format_stamp: "Formatted at 2026-08-12 14:22:51 on dist-test-slave-8w9v"
server_key: "98e738d0be8d93a1cfe96c0c543b7ac1"
server_key_iv: "8ff3a6142e47ade223b86fcdbf7213d7"
server_key_version: "encryptionkey@0"
I20260812 14:22:51.483742 10901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:51.505581 10901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:51.507687 10901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:51.507871 10901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:51.508165 10901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:51.508533 10901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:51.508594 10901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:51.508641 10901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:51.508665 10901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:51.516927 10901 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:40791
I20260812 14:22:51.516983 11050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:40791 every 8 connection(s)
I20260812 14:22:51.517325 10901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:22:51 dist-test-slave-8w9v krb5kdc[10464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544571, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:51.525094 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 10901
I20260812 14:22:51.525202 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:22:51.525439 5712 external_mini_cluster.cc:1428] Setting key b2cd12fa94a7b98be5c346267e1150eb
I20260812 14:22:51.527278 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.518190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:55033 (local address 127.5.148.62:46653)
0812 14:22:51.518265 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:51.518269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:51.518481 (+ 212us) server_negotiation.cc:407] Connection header received
0812 14:22:51.519060 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:51.519061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:51.519072 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:51.519136 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:51.520165 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.520660 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.521760 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.521963 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.522030 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.524254 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:51.524255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:51.524256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:51.524262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:51.525804 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:51.526384 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:51.526386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:51.526386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:51.526454 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:51.526716 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:51.526716 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:51.526717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:51.526829 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:51.526890 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:51.527145 (+ 255us) server_negotiation.cc:299] Negotiation successful
0812 14:22:51.527212 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 14:22:51.527560 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.518154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:46653 (local address 127.5.148.3:55033)
0812 14:22:51.518337 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:51.518385 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:51.518963 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:51.519196 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:51.519200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:51.519476 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:51.520029 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.520035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.520749 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.520750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.521645 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.521648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.522052 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.522053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.522157 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.522653 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:51.522661 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:51.524157 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:51.525924 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:51.525933 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:51.525935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:51.526293 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:51.526525 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:51.526526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:51.526527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:51.526656 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:51.526894 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:51.526899 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:51.527032 (+ 133us) client_negotiation.cc:769] Sending connection context
0812 14:22:51.527117 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 14:22:51.527242 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260812 14:22:51.528110 11051 heartbeater.cc:347] Connected to a master server at 127.5.148.62:46653
I20260812 14:22:51.528256 11051 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:51.528393 11051 heartbeater.cc:511] Master 127.5.148.62:46653 requested a full tablet report, sending...
I20260812 14:22:51.528757 10516 ts_manager.cc:194] Registered new tserver with Master: 5794f72b7716440eaaf5321e22bfdf45 (127.5.148.3:40791)
I20260812 14:22:51.529208 10516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:55033
I20260812 14:22:51.537472 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:22:51.544773 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.538798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.5.148.62:46653)
0812 14:22:51.538861 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:51.538862 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:51.538925 (+ 63us) server_negotiation.cc:407] Connection header received
0812 14:22:51.538993 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:51.538994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:51.539002 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:51.539044 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:51.539678 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.540208 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.540925 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.541113 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.541176 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.542326 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:51.542327 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:51.542328 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:51.542334 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:51.543772 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:51.544093 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:51.544094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:51.544094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:51.544168 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:51.544339 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:51.544340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:51.544340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:51.544486 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:51.544562 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:51.544651 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 14:22:51.544710 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 14:22:51.546393 10516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:22:51.548223 10516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:22:51.555033 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.550559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:43241 (local address 127.0.0.1:43374)
0812 14:22:51.551203 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:51.551214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:51.551274 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:51.551587 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:51.551587 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:51.551590 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:51.551644 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.551645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.552903 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.552904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.553845 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.553848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.554731 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.554732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.554792 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.554805 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 14:22:51.554870 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 14:22:51.554944 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260812 14:22:51.555042 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.550705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.5.148.1:43241)
0812 14:22:51.551038 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:51.551042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:51.551265 (+ 223us) server_negotiation.cc:407] Connection header received
0812 14:22:51.551395 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:51.551398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:51.551416 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:51.551471 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:51.551730 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.552804 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.553937 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.554640 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.554726 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.554788 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:51.554886 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 14:22:51.554945 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":153,"threads_started":1}
I20260812 14:22:51.555184 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.550559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:45801 (local address 127.0.0.1:43176)
0812 14:22:51.550727 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:51.550773 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:51.550870 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:51.551185 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:51.551188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:51.551191 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:51.551264 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.551266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.552641 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.552642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.554069 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.554072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.554942 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.554942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.555012 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.555015 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:22:51.555068 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 14:22:51.555109 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":117,"spinlock_wait_cycles":896,"thread_start_us":100,"threads_started":1}
I20260812 14:22:51.555459 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.550780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.5.148.2:45801)
0812 14:22:51.550984 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:51.550989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:51.551001 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:22:51.551046 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:51.551049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:51.551069 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:51.551166 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:51.551363 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.552548 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.554164 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.554852 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.554923 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.554975 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:51.555322 (+ 347us) server_negotiation.cc:299] Negotiation successful
0812 14:22:51.555369 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":87,"threads_started":1}
I20260812 14:22:51.555970 10665 tablet_service.cc:1467] Processing CreateTablet for tablet 14389ce19ea945d9bef87d66a4da3471 (DEFAULT_TABLE table=test-table [id=774b30e514cc45cf98ce985558c71509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:51.556149 10817 tablet_service.cc:1467] Processing CreateTablet for tablet 14389ce19ea945d9bef87d66a4da3471 (DEFAULT_TABLE table=test-table [id=774b30e514cc45cf98ce985558c71509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:51.556422 10665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14389ce19ea945d9bef87d66a4da3471. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:51.556483 10817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14389ce19ea945d9bef87d66a4da3471. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:51.556846 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.550763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:40791 (local address 127.0.0.1:32954)
0812 14:22:51.551274 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:51.551284 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:51.551331 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:51.551615 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:51.551616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:51.551619 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:51.551647 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.551648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.552613 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.552615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.554069 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.554072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.556591 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.556592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.556661 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.556665 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:22:51.556732 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:22:51.556779 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":39,"thread_start_us":69,"threads_started":1}
I20260812 14:22:51.556887 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.550990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.5.148.3:40791)
0812 14:22:51.551203 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:51.551207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:51.551403 (+ 196us) server_negotiation.cc:407] Connection header received
0812 14:22:51.551448 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:51.551450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:51.551465 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:51.551532 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:51.551745 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.552510 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.555012 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.556394 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.556498 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.556558 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:51.556762 (+ 204us) server_negotiation.cc:299] Negotiation successful
0812 14:22:51.556803 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":117,"threads_started":1}
I20260812 14:22:51.560316 11074 tablet_bootstrap.cc:492] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc: Bootstrap starting.
I20260812 14:22:51.560653 11075 tablet_bootstrap.cc:492] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836: Bootstrap starting.
I20260812 14:22:51.561028 10973 tablet_service.cc:1467] Processing CreateTablet for tablet 14389ce19ea945d9bef87d66a4da3471 (DEFAULT_TABLE table=test-table [id=774b30e514cc45cf98ce985558c71509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:51.561329 11074 tablet_bootstrap.cc:654] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:51.561424 10973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14389ce19ea945d9bef87d66a4da3471. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:51.561474 11075 tablet_bootstrap.cc:654] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:51.562273 11075 log.cc:824] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:51.562674 11074 log.cc:824] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:51.563199 11075 tablet_bootstrap.cc:492] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836: No bootstrap required, opened a new log
I20260812 14:22:51.563339 11075 ts_tablet_manager.cc:1397] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:22:51.563400 11074 tablet_bootstrap.cc:492] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc: No bootstrap required, opened a new log
I20260812 14:22:51.563495 11074 ts_tablet_manager.cc:1397] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:22:51.565588 11078 tablet_bootstrap.cc:492] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45: Bootstrap starting.
I20260812 14:22:51.565443 11074 raft_consensus.cc:348] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345638ef66c046b58fdb128baa2ad836" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45801 } } peers { permanent_uuid: "f7141f1c555a4224b38b80eb81d61cbc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43241 } } peers { permanent_uuid: "5794f72b7716440eaaf5321e22bfdf45" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40791 } }
I20260812 14:22:51.565696 11074 raft_consensus.cc:374] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:51.565740 11074 raft_consensus.cc:729] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7141f1c555a4224b38b80eb81d61cbc, State: Initialized, Role: FOLLOWER
I20260812 14:22:51.565864 11074 consensus_queue.cc:260] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345638ef66c046b58fdb128baa2ad836" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45801 } } peers { permanent_uuid: "f7141f1c555a4224b38b80eb81d61cbc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43241 } } peers { permanent_uuid: "5794f72b7716440eaaf5321e22bfdf45" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40791 } }
I20260812 14:22:51.566183 10741 heartbeater.cc:503] Master 127.5.148.62:46653 was elected leader, sending a full tablet report...
I20260812 14:22:51.566231 11075 raft_consensus.cc:348] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345638ef66c046b58fdb128baa2ad836" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45801 } } peers { permanent_uuid: "f7141f1c555a4224b38b80eb81d61cbc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43241 } } peers { permanent_uuid: "5794f72b7716440eaaf5321e22bfdf45" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40791 } }
I20260812 14:22:51.566373 11075 raft_consensus.cc:374] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:51.566190 11074 ts_tablet_manager.cc:1428] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:22:51.566738 11078 tablet_bootstrap.cc:654] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:51.566404 11075 raft_consensus.cc:729] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345638ef66c046b58fdb128baa2ad836, State: Initialized, Role: FOLLOWER
I20260812 14:22:51.567039 11075 consensus_queue.cc:260] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345638ef66c046b58fdb128baa2ad836" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45801 } } peers { permanent_uuid: "f7141f1c555a4224b38b80eb81d61cbc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43241 } } peers { permanent_uuid: "5794f72b7716440eaaf5321e22bfdf45" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40791 } }
I20260812 14:22:51.567356 11075 ts_tablet_manager.cc:1428] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260812 14:22:51.567612 10896 heartbeater.cc:503] Master 127.5.148.62:46653 was elected leader, sending a full tablet report...
I20260812 14:22:51.567602 11078 log.cc:824] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:51.568418 11078 tablet_bootstrap.cc:492] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45: No bootstrap required, opened a new log
I20260812 14:22:51.568506 11078 ts_tablet_manager.cc:1397] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:22:51.570189 11078 raft_consensus.cc:348] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345638ef66c046b58fdb128baa2ad836" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45801 } } peers { permanent_uuid: "f7141f1c555a4224b38b80eb81d61cbc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43241 } } peers { permanent_uuid: "5794f72b7716440eaaf5321e22bfdf45" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40791 } }
I20260812 14:22:51.570340 11078 raft_consensus.cc:374] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:51.570389 11078 raft_consensus.cc:729] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5794f72b7716440eaaf5321e22bfdf45, State: Initialized, Role: FOLLOWER
I20260812 14:22:51.570510 11078 consensus_queue.cc:260] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345638ef66c046b58fdb128baa2ad836" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45801 } } peers { permanent_uuid: "f7141f1c555a4224b38b80eb81d61cbc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43241 } } peers { permanent_uuid: "5794f72b7716440eaaf5321e22bfdf45" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40791 } }
I20260812 14:22:51.570777 11078 ts_tablet_manager.cc:1428] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 14:22:51.570825 11051 heartbeater.cc:503] Master 127.5.148.62:46653 was elected leader, sending a full tablet report...
W20260812 14:22:51.583844 10897 tablet.cc:2369] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:22:51.669139 10742 tablet.cc:2369] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:22:51.675431 11079 raft_consensus.cc:482] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:22:51.675597 11079 raft_consensus.cc:504] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345638ef66c046b58fdb128baa2ad836" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45801 } } peers { permanent_uuid: "f7141f1c555a4224b38b80eb81d61cbc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43241 } } peers { permanent_uuid: "5794f72b7716440eaaf5321e22bfdf45" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40791 } }
I20260812 14:22:51.676002 11079 leader_election.cc:288] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 345638ef66c046b58fdb128baa2ad836 (127.5.148.2:45801), 5794f72b7716440eaaf5321e22bfdf45 (127.5.148.3:40791)
I20260812 14:22:51.679201 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.676566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:35625 (local address 127.5.148.3:40791)
0812 14:22:51.676641 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:51.676643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:51.676654 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:22:51.676732 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:51.676733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:51.676745 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:51.676800 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:51.676969 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.677427 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.678434 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.678824 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.678922 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.678933 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:51.679092 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 14:22:51.679133 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 14:22:51.679210 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.676451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:40791 (local address 127.5.148.1:35625)
0812 14:22:51.676567 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:51.676582 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:51.676663 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:51.676801 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:51.676802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:51.676804 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:51.676893 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.676894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.677523 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.677524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.678324 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.678326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.678917 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.678918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.679002 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.679006 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:22:51.679077 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 14:22:51.679120 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260812 14:22:51.679399 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.676404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:33811 (local address 127.5.148.2:45801)
0812 14:22:51.676459 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:51.676461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:51.676473 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:22:51.676578 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:51.676579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:51.676593 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:51.676640 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:51.676866 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.677333 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.678426 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.678827 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.678921 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.678934 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:51.679249 (+ 315us) server_negotiation.cc:299] Negotiation successful
0812 14:22:51.679300 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:22:51.679530 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.676306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:45801 (local address 127.5.148.1:33811)
0812 14:22:51.676381 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:51.676397 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:51.676474 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:51.676661 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:51.676661 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:51.676663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:51.676764 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.676767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.677436 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.677437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.678327 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:51.678330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.678922 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.678923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:51.679075 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.679079 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:22:51.679138 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:22:51.679180 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 14:22:51.679737 10996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14389ce19ea945d9bef87d66a4da3471" candidate_uuid: "f7141f1c555a4224b38b80eb81d61cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5794f72b7716440eaaf5321e22bfdf45" is_pre_election: true
I20260812 14:22:51.679917 10996 raft_consensus.cc:2454] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7141f1c555a4224b38b80eb81d61cbc in term 0.
I20260812 14:22:51.679943 10842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14389ce19ea945d9bef87d66a4da3471" candidate_uuid: "f7141f1c555a4224b38b80eb81d61cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345638ef66c046b58fdb128baa2ad836" is_pre_election: true
I20260812 14:22:51.680148 10842 raft_consensus.cc:2454] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7141f1c555a4224b38b80eb81d61cbc in term 0.
I20260812 14:22:51.680284 10607 leader_election.cc:302] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5794f72b7716440eaaf5321e22bfdf45, f7141f1c555a4224b38b80eb81d61cbc; no voters:
I20260812 14:22:51.680500 11079 raft_consensus.cc:2781] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:22:51.680563 11079 raft_consensus.cc:482] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:22:51.680600 11079 raft_consensus.cc:3037] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:51.681442 11079 raft_consensus.cc:504] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345638ef66c046b58fdb128baa2ad836" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45801 } } peers { permanent_uuid: "f7141f1c555a4224b38b80eb81d61cbc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43241 } } peers { permanent_uuid: "5794f72b7716440eaaf5321e22bfdf45" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40791 } }
I20260812 14:22:51.681600 11079 leader_election.cc:288] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [CANDIDATE]: Term 1 election: Requested vote from peers 345638ef66c046b58fdb128baa2ad836 (127.5.148.2:45801), 5794f72b7716440eaaf5321e22bfdf45 (127.5.148.3:40791)
I20260812 14:22:51.681753 10842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14389ce19ea945d9bef87d66a4da3471" candidate_uuid: "f7141f1c555a4224b38b80eb81d61cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345638ef66c046b58fdb128baa2ad836"
I20260812 14:22:51.681833 10842 raft_consensus.cc:3037] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:51.681789 10996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14389ce19ea945d9bef87d66a4da3471" candidate_uuid: "f7141f1c555a4224b38b80eb81d61cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5794f72b7716440eaaf5321e22bfdf45"
I20260812 14:22:51.681895 10996 raft_consensus.cc:3037] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:51.682780 10842 raft_consensus.cc:2454] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7141f1c555a4224b38b80eb81d61cbc in term 1.
I20260812 14:22:51.682870 10996 raft_consensus.cc:2454] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7141f1c555a4224b38b80eb81d61cbc in term 1.
I20260812 14:22:51.683022 10607 leader_election.cc:302] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5794f72b7716440eaaf5321e22bfdf45, f7141f1c555a4224b38b80eb81d61cbc; no voters:
I20260812 14:22:51.683151 11079 raft_consensus.cc:2781] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:51.683337 11079 raft_consensus.cc:686] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [term 1 LEADER]: Becoming Leader. State: Replica: f7141f1c555a4224b38b80eb81d61cbc, State: Running, Role: LEADER
I20260812 14:22:51.683480 11079 consensus_queue.cc:237] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345638ef66c046b58fdb128baa2ad836" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45801 } } peers { permanent_uuid: "f7141f1c555a4224b38b80eb81d61cbc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43241 } } peers { permanent_uuid: "5794f72b7716440eaaf5321e22bfdf45" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40791 } }
I20260812 14:22:51.684453 10516 catalog_manager.cc:5495] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc reported cstate change: term changed from 0 to 1, leader changed from <none> to f7141f1c555a4224b38b80eb81d61cbc (127.5.148.1). New cstate: current_term: 1 leader_uuid: "f7141f1c555a4224b38b80eb81d61cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345638ef66c046b58fdb128baa2ad836" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7141f1c555a4224b38b80eb81d61cbc" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5794f72b7716440eaaf5321e22bfdf45" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40791 } health_report { overall_health: UNKNOWN } } }
I20260812 14:22:51.700047 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.697416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.5.148.1:43241)
0812 14:22:51.697486 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:51.697488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:51.697498 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:22:51.697596 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:51.697597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:51.697623 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:51.697712 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:22:51.697887 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.698378 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.699155 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:51.699374 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:51.699459 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:51.699634 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:22:51.699754 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:22:51.699847 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:51.699898 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 14:22:51.699929 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:22:51.702031 10996 raft_consensus.cc:1261] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45 [term 1 FOLLOWER]: Refusing update from remote peer f7141f1c555a4224b38b80eb81d61cbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:51.702067 10842 raft_consensus.cc:1261] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836 [term 1 FOLLOWER]: Refusing update from remote peer f7141f1c555a4224b38b80eb81d61cbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:51.702402 11079 consensus_queue.cc:1035] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5794f72b7716440eaaf5321e22bfdf45" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:51.702561 11084 consensus_queue.cc:1035] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "345638ef66c046b58fdb128baa2ad836" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:51.708336 11088 mvcc.cc:205] Tried to move back new op lower bound from 7317686565688705024 to 7317686565616295936. Current Snapshot: MvccSnapshot[applied={T|T < 7317686565688705024}]
I20260812 14:22:51.710649 10516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:51.710760 10516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:51.711644 10516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e854119cf4e4821afba3c9a65db80d2: Sending DeleteTablet for 3 replicas of tablet 14389ce19ea945d9bef87d66a4da3471
I20260812 14:22:51.711992 10817 tablet_service.cc:1514] Processing DeleteTablet for tablet 14389ce19ea945d9bef87d66a4da3471 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:51 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:43176
I20260812 14:22:51.712047 10665 tablet_service.cc:1514] Processing DeleteTablet for tablet 14389ce19ea945d9bef87d66a4da3471 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:51 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:43374
I20260812 14:22:51.712172 10973 tablet_service.cc:1514] Processing DeleteTablet for tablet 14389ce19ea945d9bef87d66a4da3471 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:51 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:32954
I20260812 14:22:51.712417 11102 tablet_replica.cc:330] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45: stopping tablet replica
I20260812 14:22:51.712527 11101 tablet_replica.cc:330] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc: stopping tablet replica
I20260812 14:22:51.712584 11102 raft_consensus.cc:2229] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:51.712704 11101 raft_consensus.cc:2229] T 14389ce19ea945d9bef87d66a4da3471 P f7141f1c555a4224b38b80eb81d61cbc [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:22:51.712708 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 10591
I20260812 14:22:51.712743 11102 raft_consensus.cc:2258] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:51.713431 11102 ts_tablet_manager.cc:1905] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:51.715574 11102 ts_tablet_manager.cc:1918] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:22:51.715771 11102 log.cc:1196] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/14389ce19ea945d9bef87d66a4da3471
I20260812 14:22:51.716348 11102 ts_tablet_manager.cc:1939] T 14389ce19ea945d9bef87d66a4da3471 P 5794f72b7716440eaaf5321e22bfdf45: Deleting consensus metadata
I20260812 14:22:51.717141 10497 catalog_manager.cc:4841] TS 5794f72b7716440eaaf5321e22bfdf45 (127.5.148.3:40791): tablet 14389ce19ea945d9bef87d66a4da3471 (table test-table [id=774b30e514cc45cf98ce985558c71509]) successfully deleted
I20260812 14:22:51.720320 11100 tablet_replica.cc:330] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836: stopping tablet replica
I20260812 14:22:51.720510 11100 raft_consensus.cc:2229] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:51.720666 11100 raft_consensus.cc:2258] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:51.721143 11100 ts_tablet_manager.cc:1905] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:51.722805 11100 ts_tablet_manager.cc:1918] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:22:51.722918 11100 log.cc:1196] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/14389ce19ea945d9bef87d66a4da3471
I20260812 14:22:51.723178 11100 ts_tablet_manager.cc:1939] T 14389ce19ea945d9bef87d66a4da3471 P 345638ef66c046b58fdb128baa2ad836: Deleting consensus metadata
W20260812 14:22:51.723608 10499 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:43241 (error 108)
I20260812 14:22:51.723817 10497 catalog_manager.cc:4841] TS 345638ef66c046b58fdb128baa2ad836 (127.5.148.2:45801): tablet 14389ce19ea945d9bef87d66a4da3471 (table test-table [id=774b30e514cc45cf98ce985558c71509]) successfully deleted
I20260812 14:22:51.723835 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 10746
I20260812 14:22:51.725760 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.725626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:43241 (local address 127.0.0.1:43388)
0812 14:22:51.725680 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:51.725719 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 14:22:51.725874 10499 catalog_manager.cc:4584] TS f7141f1c555a4224b38b80eb81d61cbc (127.5.148.1:43241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14389ce19ea945d9bef87d66a4da3471: Network error: Client connection negotiation failed: client connection to 127.5.148.1:43241: connect: Connection refused (error 111)
I20260812 14:22:51.737723 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 10901
I20260812 14:22:51.754523 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 10481
2026-08-12T14:22:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2200 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T14:22:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:51Z Disabled control of system clock
I20260812 14:22:51.787117 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:35053
--webserver_interface=127.5.148.62
--webserver_port=0
--builtin_ntp_servers=127.5.148.20:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:35053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 14:22:51.871809 11107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:51.872032 11107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:51.872056 11107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:51.873229 11107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:51.873270 11107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:51.873283 11107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:51.873296 11107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:51.873308 11107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:22:51.874645 11107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:35053
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:35053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:51.874863 11107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:51.875121 11107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:51.880241 11112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:51.880259 11113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:51.880244 11116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:51.880774 11107 server_base.cc:1034] running on GCE node
I20260812 14:22:51.881033 11107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:51.881299 11107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:51.882447 11107 hybrid_clock.cc:648] HybridClock initialized: now 1786544571882432 us; error 29 us; skew 500 ppm
I20260812 14:22:51.883729 11107 webserver.cc:460] Webserver started at http://127.5.148.62:35695/ using document root <none> and password file <none>
I20260812 14:22:51.884013 11107 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:51.884063 11107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:51.884248 11107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:51.885401 11107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "aa052ca123f042f3aceaebede810040b"
format_stamp: "Formatted at 2026-08-12 14:22:51 on dist-test-slave-8w9v"
server_key: "f73354bb6819e5904b613ee2732c0d5d"
server_key_iv: "3211b007520e3f10bfd307372387af90"
server_key_version: "encryptionkey@0"
I20260812 14:22:51.885778 11107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "aa052ca123f042f3aceaebede810040b"
format_stamp: "Formatted at 2026-08-12 14:22:51 on dist-test-slave-8w9v"
server_key: "f73354bb6819e5904b613ee2732c0d5d"
server_key_iv: "3211b007520e3f10bfd307372387af90"
server_key_version: "encryptionkey@0"
I20260812 14:22:51.887126 11107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:22:51.887863 11123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:51.888059 11107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:22:51.888204 11107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "aa052ca123f042f3aceaebede810040b"
format_stamp: "Formatted at 2026-08-12 14:22:51 on dist-test-slave-8w9v"
server_key: "f73354bb6819e5904b613ee2732c0d5d"
server_key_iv: "3211b007520e3f10bfd307372387af90"
server_key_version: "encryptionkey@0"
I20260812 14:22:51.888289 11107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:51.893028 11107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:51.893289 11107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:51.893399 11107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:51.897384 11107 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:35053
I20260812 14:22:51.897419 11186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:35053 every 8 connection(s)
I20260812 14:22:51.897758 11107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:51.898429 11187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:51.901350 11187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b: Bootstrap starting.
I20260812 14:22:51.901907 11187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:51.902060 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 11107
I20260812 14:22:51.902171 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:51.902266 11187 log.cc:824] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:51.902419 5712 external_mini_cluster.cc:1428] Setting key dd197e914233cfba614b14c859062777
I20260812 14:22:51.903151 11187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b: No bootstrap required, opened a new log
I20260812 14:22:51.905110 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:51.902960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.5.148.62:35053)
0812 14:22:51.903179 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:51.903188 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:51.903224 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:22:51.903756 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:51.903766 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:51.903819 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:51.903981 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:51.904651 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:51.904655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:51.904655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:51.904661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:51.904710 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:51.904723 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:51.904822 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:51.904853 (+ 31us) server_negotiation.cc:299] Negotiation successful
0812 14:22:51.904912 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260812 14:22:51.905505 11187 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa052ca123f042f3aceaebede810040b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35053 } }
I20260812 14:22:51.905679 11187 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:51.905714 11187 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa052ca123f042f3aceaebede810040b, State: Initialized, Role: FOLLOWER
I20260812 14:22:51.905815 11187 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa052ca123f042f3aceaebede810040b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35053 } }
I20260812 14:22:51.905906 11187 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:51.905956 11187 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:51.906009 11187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:51.906862 11187 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa052ca123f042f3aceaebede810040b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35053 } }
I20260812 14:22:51.906965 11187 leader_election.cc:302] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa052ca123f042f3aceaebede810040b; no voters:
I20260812 14:22:51.907128 11192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:51.907316 11192 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [term 1 LEADER]: Becoming Leader. State: Replica: aa052ca123f042f3aceaebede810040b, State: Running, Role: LEADER
I20260812 14:22:51.907444 11192 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa052ca123f042f3aceaebede810040b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35053 } }
I20260812 14:22:51.907115 11187 leader_election.cc:288] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:51.907804 11193 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa052ca123f042f3aceaebede810040b. Latest consensus state: current_term: 1 leader_uuid: "aa052ca123f042f3aceaebede810040b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa052ca123f042f3aceaebede810040b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35053 } } }
I20260812 14:22:51.907907 11193 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:51.907977 11187 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:51.908049 11193 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa052ca123f042f3aceaebede810040b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa052ca123f042f3aceaebede810040b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35053 } } }
I20260812 14:22:51.908103 11193 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:51.908265 11195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:22:51.908861 11195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:22:51.910418 11195 catalog_manager.cc:1347] Generated new cluster ID: 8566f1cce665437a82c1d3e790143576
I20260812 14:22:51.910473 11195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:22:51.924935 11195 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:51.925148 11195 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:51.929988 11195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b: Generated new TSK 0
I20260812 14:22:51.930215 11195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:22:51.938668 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35053
--builtin_ntp_servers=127.5.148.20:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 14:22:52.025089 11212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:52.025293 11212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:52.025317 11212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:52.026522 11212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:52.026564 11212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:52.026590 11212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:22:52.027956 11212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:52.028267 11212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:52.028599 11212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:52.034416 11220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:52.034513 11223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:52.034582 11212 server_base.cc:1034] running on GCE node
W20260812 14:22:52.034631 11219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:52.034873 11212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:52.035142 11212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:52.036316 11212 hybrid_clock.cc:648] HybridClock initialized: now 1786544572036310 us; error 40 us; skew 500 ppm
I20260812 14:22:52.037694 11212 webserver.cc:460] Webserver started at http://127.5.148.1:41021/ using document root <none> and password file <none>
I20260812 14:22:52.037981 11212 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:52.038057 11212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:52.038220 11212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:52.039383 11212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "4dde00539a1240bd9958dc8b72fe90e6"
format_stamp: "Formatted at 2026-08-12 14:22:52 on dist-test-slave-8w9v"
server_key: "bb1ca06d6e69ee08dc38d23f34eda24d"
server_key_iv: "177f5bb0687c8605aebfe14436b18a02"
server_key_version: "encryptionkey@0"
I20260812 14:22:52.039795 11212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "4dde00539a1240bd9958dc8b72fe90e6"
format_stamp: "Formatted at 2026-08-12 14:22:52 on dist-test-slave-8w9v"
server_key: "bb1ca06d6e69ee08dc38d23f34eda24d"
server_key_iv: "177f5bb0687c8605aebfe14436b18a02"
server_key_version: "encryptionkey@0"
I20260812 14:22:52.041301 11212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:22:52.042034 11229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:52.042210 11212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:22:52.042321 11212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "4dde00539a1240bd9958dc8b72fe90e6"
format_stamp: "Formatted at 2026-08-12 14:22:52 on dist-test-slave-8w9v"
server_key: "bb1ca06d6e69ee08dc38d23f34eda24d"
server_key_iv: "177f5bb0687c8605aebfe14436b18a02"
server_key_version: "encryptionkey@0"
I20260812 14:22:52.042402 11212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:52.045789 11212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:52.046059 11212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:52.046200 11212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:52.046398 11212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:52.046713 11212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:52.046766 11212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:52.046806 11212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:52.046834 11212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:52.052613 11212 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:45549
I20260812 14:22:52.052695 11358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:45549 every 8 connection(s)
I20260812 14:22:52.053028 11212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:22:52.054466 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 11212
I20260812 14:22:52.054549 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:22:52.054737 5712 external_mini_cluster.cc:1428] Setting key 91368a474443c422f612f8151ec78867
I20260812 14:22:52.055938 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35053
--builtin_ntp_servers=127.5.148.20:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 14:22:52.056710 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.053793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:34283 (local address 127.5.148.62:35053)
0812 14:22:52.053845 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:52.053847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:52.054101 (+ 254us) server_negotiation.cc:407] Connection header received
0812 14:22:52.054787 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:52.054789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:52.054797 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:52.054865 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:52.056419 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:52.056421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:52.056421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:52.056427 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:52.056448 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:52.056454 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:52.056538 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:52.056588 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 14:22:52.056605 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:22:52.056828 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.053756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35053 (local address 127.5.148.1:34283)
0812 14:22:52.053973 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:52.054007 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:52.054632 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:52.054937 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:52.054941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:52.054950 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:52.056262 (+ 1312us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:22:52.056262 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:52.056292 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:22:52.056321 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:52.056534 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:52.056536 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:52.056539 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:22:52.056590 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 14:22:52.056638 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":95,"threads_started":1}
I20260812 14:22:52.057380 11360 heartbeater.cc:347] Connected to a master server at 127.5.148.62:35053
I20260812 14:22:52.057512 11360 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:52.057663 11360 heartbeater.cc:511] Master 127.5.148.62:35053 requested a full tablet report, sending...
I20260812 14:22:52.058166 11141 ts_manager.cc:194] Registered new tserver with Master: 4dde00539a1240bd9958dc8b72fe90e6 (127.5.148.1:45549)
W20260812 14:22:52.148061 11363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:52.148362 11363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:52.148387 11363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:52.149609 11363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:52.149649 11363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:52.149699 11363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:22:52.151068 11363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:52.151330 11363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:52.151648 11363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:52.157022 11372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:52.157084 11368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:52.157032 11370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:52.157244 11363 server_base.cc:1034] running on GCE node
I20260812 14:22:52.157513 11363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:52.157763 11363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:52.158931 11363 hybrid_clock.cc:648] HybridClock initialized: now 1786544572158908 us; error 38 us; skew 500 ppm
I20260812 14:22:52.160398 11363 webserver.cc:460] Webserver started at http://127.5.148.2:35557/ using document root <none> and password file <none>
I20260812 14:22:52.160658 11363 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:52.160730 11363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:52.160892 11363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:52.162068 11363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "1e82a2d3665f4c1b92efa804edc060ba"
format_stamp: "Formatted at 2026-08-12 14:22:52 on dist-test-slave-8w9v"
server_key: "041fc372325590b85d4fd79fd4b0d3b7"
server_key_iv: "2178fadd240c1a1c6180b7e0461bbe54"
server_key_version: "encryptionkey@0"
I20260812 14:22:52.162504 11363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "1e82a2d3665f4c1b92efa804edc060ba"
format_stamp: "Formatted at 2026-08-12 14:22:52 on dist-test-slave-8w9v"
server_key: "041fc372325590b85d4fd79fd4b0d3b7"
server_key_iv: "2178fadd240c1a1c6180b7e0461bbe54"
server_key_version: "encryptionkey@0"
I20260812 14:22:52.163933 11363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:22:52.164763 11378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:52.164971 11363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:22:52.165081 11363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "1e82a2d3665f4c1b92efa804edc060ba"
format_stamp: "Formatted at 2026-08-12 14:22:52 on dist-test-slave-8w9v"
server_key: "041fc372325590b85d4fd79fd4b0d3b7"
server_key_iv: "2178fadd240c1a1c6180b7e0461bbe54"
server_key_version: "encryptionkey@0"
I20260812 14:22:52.165167 11363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:52.168622 11363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:52.168955 11363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:52.169111 11363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:52.169327 11363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:52.169653 11363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:52.169708 11363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:52.169751 11363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:52.169781 11363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:52.177273 11363 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:34717
I20260812 14:22:52.177455 11515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:34717 every 8 connection(s)
I20260812 14:22:52.177726 11363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:22:52.181660 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.178750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:52461 (local address 127.5.148.62:35053)
0812 14:22:52.178821 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:52.178823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:52.179183 (+ 360us) server_negotiation.cc:407] Connection header received
0812 14:22:52.179925 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:52.179927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:52.179934 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:52.179995 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:52.181391 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:52.181393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:52.181393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:52.181398 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:52.181415 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:52.181419 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:52.181480 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:52.181564 (+ 84us) server_negotiation.cc:299] Negotiation successful
0812 14:22:52.181583 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 14:22:52.181941 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.178715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35053 (local address 127.5.148.2:52461)
0812 14:22:52.179039 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:52.179086 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:52.179795 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:52.180075 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:52.180079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:52.180090 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:52.181226 (+ 1136us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:22:52.181226 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:52.181258 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:22:52.181288 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:52.181479 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:52.181485 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:52.181488 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:22:52.181578 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 14:22:52.181659 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":191,"threads_started":1}
I20260812 14:22:52.182416 11516 heartbeater.cc:347] Connected to a master server at 127.5.148.62:35053
I20260812 14:22:52.182502 11516 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:52.182693 11516 heartbeater.cc:511] Master 127.5.148.62:35053 requested a full tablet report, sending...
I20260812 14:22:52.182765 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 11363
I20260812 14:22:52.182861 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:22:52.183071 5712 external_mini_cluster.cc:1428] Setting key 2e35e958187fba927765fdb5fe9af99d
I20260812 14:22:52.183163 11141 ts_manager.cc:194] Registered new tserver with Master: 1e82a2d3665f4c1b92efa804edc060ba (127.5.148.2:34717)
I20260812 14:22:52.184105 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35053
--builtin_ntp_servers=127.5.148.20:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 14:22:52.277717 11519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:52.277952 11519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:52.277980 11519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:52.279320 11519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:52.279372 11519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:52.279431 11519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:22:52.281167 11519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:52.281471 11519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:52.281940 11519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:52.287562 11524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:52.287768 11526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:52.287639 11528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:52.287863 11519 server_base.cc:1034] running on GCE node
I20260812 14:22:52.288057 11519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:52.288374 11519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:52.289551 11519 hybrid_clock.cc:648] HybridClock initialized: now 1786544572289531 us; error 41 us; skew 500 ppm
I20260812 14:22:52.290868 11519 webserver.cc:460] Webserver started at http://127.5.148.3:38489/ using document root <none> and password file <none>
I20260812 14:22:52.291121 11519 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:52.291168 11519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:52.291344 11519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:52.292536 11519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "c62adc3b00c449baa2d19f484f6278bf"
format_stamp: "Formatted at 2026-08-12 14:22:52 on dist-test-slave-8w9v"
server_key: "6ba3013f5bfa0c6ceac6dc82cae4e0b4"
server_key_iv: "a270d56325489c842efb9eee2c6e5641"
server_key_version: "encryptionkey@0"
I20260812 14:22:52.292912 11519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "c62adc3b00c449baa2d19f484f6278bf"
format_stamp: "Formatted at 2026-08-12 14:22:52 on dist-test-slave-8w9v"
server_key: "6ba3013f5bfa0c6ceac6dc82cae4e0b4"
server_key_iv: "a270d56325489c842efb9eee2c6e5641"
server_key_version: "encryptionkey@0"
I20260812 14:22:52.294293 11519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:22:52.295085 11535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:52.295281 11519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:52.295377 11519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "c62adc3b00c449baa2d19f484f6278bf"
format_stamp: "Formatted at 2026-08-12 14:22:52 on dist-test-slave-8w9v"
server_key: "6ba3013f5bfa0c6ceac6dc82cae4e0b4"
server_key_iv: "a270d56325489c842efb9eee2c6e5641"
server_key_version: "encryptionkey@0"
I20260812 14:22:52.295466 11519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:52.299228 11519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:52.299518 11519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:52.299649 11519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:52.299871 11519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:52.300264 11519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:52.300302 11519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:52.300323 11519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:52.300369 11519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:52.308905 11519 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:40329
I20260812 14:22:52.308964 11671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:40329 every 8 connection(s)
I20260812 14:22:52.309474 11519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:22:52.310451 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 11519
I20260812 14:22:52.310556 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:22:52.310753 5712 external_mini_cluster.cc:1428] Setting key 41892b1571d02646c0ecf6a8e0ceca9e
I20260812 14:22:52.313436 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.310545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:39795 (local address 127.5.148.62:35053)
0812 14:22:52.310608 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:52.310611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:52.310911 (+ 300us) server_negotiation.cc:407] Connection header received
0812 14:22:52.311615 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:52.311616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:52.311623 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:52.311690 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:52.313204 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:52.313205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:52.313206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:52.313210 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:52.313225 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:52.313230 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:52.313288 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:52.313368 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 14:22:52.313381 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:22:52.313637 11674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.310459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35053 (local address 127.5.148.3:39795)
0812 14:22:52.310762 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:52.310828 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:52.311499 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:52.311798 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:52.311802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:52.311812 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:52.313034 (+ 1222us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:22:52.313035 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:52.313065 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:22:52.313095 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:52.313297 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:52.313299 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:52.313302 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:22:52.313367 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 14:22:52.313418 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":88,"threads_started":1}
I20260812 14:22:52.314164 11672 heartbeater.cc:347] Connected to a master server at 127.5.148.62:35053
I20260812 14:22:52.314275 11672 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:52.314453 11672 heartbeater.cc:511] Master 127.5.148.62:35053 requested a full tablet report, sending...
I20260812 14:22:52.314821 11141 ts_manager.cc:194] Registered new tserver with Master: c62adc3b00c449baa2d19f484f6278bf (127.5.148.3:40329)
I20260812 14:22:52.322446 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:22:52.324918 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.323913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.5.148.62:35053)
0812 14:22:52.323966 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:52.323968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:52.324102 (+ 134us) server_negotiation.cc:407] Connection header received
0812 14:22:52.324177 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:52.324178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:52.324184 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:52.324243 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:52.324749 (+ 506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:52.324750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:52.324750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:52.324753 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:52.324766 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:52.324770 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:52.324817 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:52.324854 (+ 37us) server_negotiation.cc:299] Negotiation successful
0812 14:22:52.324867 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 14:22:52.326431 11141 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:22:52.327951 11141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:22:52.331923 11681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.330120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:34717 (local address 127.0.0.1:54322)
0812 14:22:52.330440 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:52.330451 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:52.330510 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:52.330711 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:52.330718 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:52.330724 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:52.331249 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:22:52.331250 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:52.331269 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:22:52.331296 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:52.331732 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:52.331740 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:52.331743 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:22:52.331822 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 14:22:52.331837 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":103,"threads_started":1}
I20260812 14:22:52.331959 11683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.330228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.5.148.2:34717)
0812 14:22:52.330540 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:52.330545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:52.330557 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:22:52.330586 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:52.330588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:52.330601 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:52.330711 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:52.331564 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:52.331568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:52.331568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:52.331594 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:52.331635 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:52.331652 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:52.331723 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:52.331827 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 14:22:52.331847 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":81,"threads_started":1}
I20260812 14:22:52.332031 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.330118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:45549 (local address 127.0.0.1:45664)
0812 14:22:52.330279 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:52.330320 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:52.330379 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:52.331076 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:52.331077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:52.331081 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:52.331505 (+ 424us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:22:52.331506 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:52.331513 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:22:52.331526 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:52.331863 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:52.331864 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:52.331865 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:22:52.331920 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 14:22:52.331934 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":59,"threads_started":1}
I20260812 14:22:52.332525 11684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.330282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.5.148.1:45549)
0812 14:22:52.330656 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:52.330660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:52.330671 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:22:52.330700 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:52.330702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:52.330711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:52.331291 (+ 580us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:52.331740 (+ 449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:52.331744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:52.331744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:52.331748 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:52.331780 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:52.331797 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:52.331857 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:52.331933 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 14:22:52.331949 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":99,"threads_started":1}
I20260812 14:22:52.332763 11685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.330326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:40329 (local address 127.0.0.1:45076)
0812 14:22:52.330588 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:52.330599 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:52.330640 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:52.331441 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:52.331442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:52.331445 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:52.332362 (+ 917us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:22:52.332363 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:52.332370 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:22:52.332382 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:52.332639 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:52.332640 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:52.332641 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:22:52.332690 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:22:52.332703 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":198,"threads_started":1}
I20260812 14:22:52.332971 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.330661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.5.148.3:40329)
0812 14:22:52.330950 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:52.330955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:52.330965 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:22:52.330990 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:52.330992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:52.331001 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:52.331133 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:52.332481 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:52.332488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:52.332489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:52.332493 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:52.332529 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:52.332552 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:52.332609 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:52.332865 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 14:22:52.332882 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":173,"threads_started":1}
I20260812 14:22:52.332858 11436 tablet_service.cc:1467] Processing CreateTablet for tablet d5c992efa96140d5a937d7ede28f42bc (DEFAULT_TABLE table=test-table [id=bff85cb9ce6842acb8e87363104cceb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:52.333171 11282 tablet_service.cc:1467] Processing CreateTablet for tablet d5c992efa96140d5a937d7ede28f42bc (DEFAULT_TABLE table=test-table [id=bff85cb9ce6842acb8e87363104cceb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:52.333325 11436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c992efa96140d5a937d7ede28f42bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:52.333503 11282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c992efa96140d5a937d7ede28f42bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:52.333483 11592 tablet_service.cc:1467] Processing CreateTablet for tablet d5c992efa96140d5a937d7ede28f42bc (DEFAULT_TABLE table=test-table [id=bff85cb9ce6842acb8e87363104cceb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:22:52.333730 11592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c992efa96140d5a937d7ede28f42bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:52.336710 11687 tablet_bootstrap.cc:492] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf: Bootstrap starting.
I20260812 14:22:52.336948 11693 tablet_bootstrap.cc:492] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6: Bootstrap starting.
I20260812 14:22:52.336977 11692 tablet_bootstrap.cc:492] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba: Bootstrap starting.
I20260812 14:22:52.337669 11687 tablet_bootstrap.cc:654] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:52.337738 11693 tablet_bootstrap.cc:654] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:52.337809 11692 tablet_bootstrap.cc:654] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:52.338236 11693 log.cc:824] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:52.338253 11692 log.cc:824] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:52.338238 11687 log.cc:824] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:52.339169 11693 tablet_bootstrap.cc:492] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6: No bootstrap required, opened a new log
I20260812 14:22:52.339224 11693 ts_tablet_manager.cc:1397] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:22:52.339367 11687 tablet_bootstrap.cc:492] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf: No bootstrap required, opened a new log
I20260812 14:22:52.339448 11687 ts_tablet_manager.cc:1397] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:22:52.340595 11692 tablet_bootstrap.cc:492] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba: No bootstrap required, opened a new log
I20260812 14:22:52.340667 11692 ts_tablet_manager.cc:1397] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 14:22:52.341418 11693 raft_consensus.cc:348] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde00539a1240bd9958dc8b72fe90e6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45549 } } peers { permanent_uuid: "1e82a2d3665f4c1b92efa804edc060ba" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34717 } } peers { permanent_uuid: "c62adc3b00c449baa2d19f484f6278bf" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40329 } }
I20260812 14:22:52.341604 11693 raft_consensus.cc:374] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:52.341648 11693 raft_consensus.cc:729] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dde00539a1240bd9958dc8b72fe90e6, State: Initialized, Role: FOLLOWER
I20260812 14:22:52.341792 11693 consensus_queue.cc:260] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde00539a1240bd9958dc8b72fe90e6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45549 } } peers { permanent_uuid: "1e82a2d3665f4c1b92efa804edc060ba" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34717 } } peers { permanent_uuid: "c62adc3b00c449baa2d19f484f6278bf" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40329 } }
I20260812 14:22:52.342126 11693 ts_tablet_manager.cc:1428] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:22:52.342273 11360 heartbeater.cc:503] Master 127.5.148.62:35053 was elected leader, sending a full tablet report...
I20260812 14:22:52.342485 11687 raft_consensus.cc:348] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde00539a1240bd9958dc8b72fe90e6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45549 } } peers { permanent_uuid: "1e82a2d3665f4c1b92efa804edc060ba" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34717 } } peers { permanent_uuid: "c62adc3b00c449baa2d19f484f6278bf" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40329 } }
I20260812 14:22:52.342803 11687 raft_consensus.cc:374] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:52.342850 11687 raft_consensus.cc:729] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c62adc3b00c449baa2d19f484f6278bf, State: Initialized, Role: FOLLOWER
I20260812 14:22:52.342986 11687 consensus_queue.cc:260] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde00539a1240bd9958dc8b72fe90e6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45549 } } peers { permanent_uuid: "1e82a2d3665f4c1b92efa804edc060ba" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34717 } } peers { permanent_uuid: "c62adc3b00c449baa2d19f484f6278bf" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40329 } }
I20260812 14:22:52.343297 11687 ts_tablet_manager.cc:1428] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260812 14:22:52.343333 11672 heartbeater.cc:503] Master 127.5.148.62:35053 was elected leader, sending a full tablet report...
I20260812 14:22:52.343732 11692 raft_consensus.cc:348] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde00539a1240bd9958dc8b72fe90e6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45549 } } peers { permanent_uuid: "1e82a2d3665f4c1b92efa804edc060ba" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34717 } } peers { permanent_uuid: "c62adc3b00c449baa2d19f484f6278bf" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40329 } }
I20260812 14:22:52.343875 11692 raft_consensus.cc:374] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:52.343923 11692 raft_consensus.cc:729] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e82a2d3665f4c1b92efa804edc060ba, State: Initialized, Role: FOLLOWER
I20260812 14:22:52.344046 11692 consensus_queue.cc:260] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde00539a1240bd9958dc8b72fe90e6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45549 } } peers { permanent_uuid: "1e82a2d3665f4c1b92efa804edc060ba" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34717 } } peers { permanent_uuid: "c62adc3b00c449baa2d19f484f6278bf" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40329 } }
I20260812 14:22:52.344399 11692 ts_tablet_manager.cc:1428] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260812 14:22:52.344496 11516 heartbeater.cc:503] Master 127.5.148.62:35053 was elected leader, sending a full tablet report...
I20260812 14:22:52.385345 11697 raft_consensus.cc:482] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:22:52.385490 11697 raft_consensus.cc:504] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde00539a1240bd9958dc8b72fe90e6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45549 } } peers { permanent_uuid: "1e82a2d3665f4c1b92efa804edc060ba" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34717 } } peers { permanent_uuid: "c62adc3b00c449baa2d19f484f6278bf" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40329 } }
I20260812 14:22:52.385811 11697 leader_election.cc:288] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e82a2d3665f4c1b92efa804edc060ba (127.5.148.2:34717), c62adc3b00c449baa2d19f484f6278bf (127.5.148.3:40329)
I20260812 14:22:52.387305 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.386048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:34717 (local address 127.5.148.1:56519)
0812 14:22:52.386113 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:52.386127 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:52.386172 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:52.386359 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:52.386360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:52.386363 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:52.386886 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:22:52.386887 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:52.386897 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:22:52.386913 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:52.387123 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:52.387124 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:52.387125 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:22:52.387203 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 14:22:52.387218 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 14:22:52.387372 11683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.386150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:56519 (local address 127.5.148.2:34717)
0812 14:22:52.386200 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:52.386203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:52.386215 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:22:52.386250 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:52.386251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:52.386257 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:52.386333 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:52.387010 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:52.387012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:52.387012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:52.387017 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:52.387038 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:52.387044 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:52.387111 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:52.387215 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 14:22:52.387236 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:22:52.387477 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.386332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:53543 (local address 127.5.148.3:40329)
0812 14:22:52.386387 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:52.386389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:52.386512 (+ 123us) server_negotiation.cc:407] Connection header received
0812 14:22:52.386587 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:52.386588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:52.386593 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:52.386669 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:52.387186 (+ 517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:52.387189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:52.387189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:52.387194 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:52.387214 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:52.387219 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:52.387276 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:52.387376 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 14:22:52.387391 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 14:22:52.387552 11700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.386185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:40329 (local address 127.5.148.1:53543)
0812 14:22:52.386440 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:52.386453 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:52.386520 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:52.386674 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:52.386675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:52.386679 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:52.387076 (+ 397us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:22:52.387077 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:52.387084 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:22:52.387098 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:52.387314 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:52.387315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:52.387316 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:22:52.387364 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:22:52.387378 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":185,"threads_started":1}
I20260812 14:22:52.387748 11461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c992efa96140d5a937d7ede28f42bc" candidate_uuid: "4dde00539a1240bd9958dc8b72fe90e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e82a2d3665f4c1b92efa804edc060ba" is_pre_election: true
I20260812 14:22:52.387884 11615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c992efa96140d5a937d7ede28f42bc" candidate_uuid: "4dde00539a1240bd9958dc8b72fe90e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c62adc3b00c449baa2d19f484f6278bf" is_pre_election: true
I20260812 14:22:52.387951 11461 raft_consensus.cc:2454] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dde00539a1240bd9958dc8b72fe90e6 in term 0.
I20260812 14:22:52.388005 11615 raft_consensus.cc:2454] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dde00539a1240bd9958dc8b72fe90e6 in term 0.
I20260812 14:22:52.388245 11233 leader_election.cc:302] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dde00539a1240bd9958dc8b72fe90e6, c62adc3b00c449baa2d19f484f6278bf; no voters:
I20260812 14:22:52.388382 11697 raft_consensus.cc:2781] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:22:52.388443 11697 raft_consensus.cc:482] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:22:52.388477 11697 raft_consensus.cc:3037] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:52.389307 11697 raft_consensus.cc:504] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde00539a1240bd9958dc8b72fe90e6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45549 } } peers { permanent_uuid: "1e82a2d3665f4c1b92efa804edc060ba" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34717 } } peers { permanent_uuid: "c62adc3b00c449baa2d19f484f6278bf" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40329 } }
I20260812 14:22:52.389446 11697 leader_election.cc:288] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1e82a2d3665f4c1b92efa804edc060ba (127.5.148.2:34717), c62adc3b00c449baa2d19f484f6278bf (127.5.148.3:40329)
I20260812 14:22:52.389645 11461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c992efa96140d5a937d7ede28f42bc" candidate_uuid: "4dde00539a1240bd9958dc8b72fe90e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e82a2d3665f4c1b92efa804edc060ba"
I20260812 14:22:52.389645 11615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c992efa96140d5a937d7ede28f42bc" candidate_uuid: "4dde00539a1240bd9958dc8b72fe90e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c62adc3b00c449baa2d19f484f6278bf"
I20260812 14:22:52.389735 11461 raft_consensus.cc:3037] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:52.389735 11615 raft_consensus.cc:3037] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:52.390947 11615 raft_consensus.cc:2454] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dde00539a1240bd9958dc8b72fe90e6 in term 1.
I20260812 14:22:52.390961 11461 raft_consensus.cc:2454] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dde00539a1240bd9958dc8b72fe90e6 in term 1.
I20260812 14:22:52.391112 11233 leader_election.cc:302] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dde00539a1240bd9958dc8b72fe90e6, c62adc3b00c449baa2d19f484f6278bf; no voters:
I20260812 14:22:52.391209 11697 raft_consensus.cc:2781] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:52.391402 11697 raft_consensus.cc:686] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [term 1 LEADER]: Becoming Leader. State: Replica: 4dde00539a1240bd9958dc8b72fe90e6, State: Running, Role: LEADER
I20260812 14:22:52.391520 11697 consensus_queue.cc:237] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde00539a1240bd9958dc8b72fe90e6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45549 } } peers { permanent_uuid: "1e82a2d3665f4c1b92efa804edc060ba" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34717 } } peers { permanent_uuid: "c62adc3b00c449baa2d19f484f6278bf" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40329 } }
I20260812 14:22:52.392488 11141 catalog_manager.cc:5495] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dde00539a1240bd9958dc8b72fe90e6 (127.5.148.1). New cstate: current_term: 1 leader_uuid: "4dde00539a1240bd9958dc8b72fe90e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde00539a1240bd9958dc8b72fe90e6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e82a2d3665f4c1b92efa804edc060ba" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c62adc3b00c449baa2d19f484f6278bf" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40329 } health_report { overall_health: UNKNOWN } } }
I20260812 14:22:52.406895 11684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.405810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.5.148.1:45549)
0812 14:22:52.405876 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:52.405878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:52.405909 (+ 31us) server_negotiation.cc:407] Connection header received
0812 14:22:52.405954 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:52.405955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:52.405969 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:52.406013 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:52.406673 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:52.406674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:52.406674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:52.406677 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:52.406688 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:52.406692 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:52.406733 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:52.406774 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 14:22:52.406784 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 14:22:52.408972 11615 raft_consensus.cc:1261] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf [term 1 FOLLOWER]: Refusing update from remote peer 4dde00539a1240bd9958dc8b72fe90e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:52.409081 11461 raft_consensus.cc:1261] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba [term 1 FOLLOWER]: Refusing update from remote peer 4dde00539a1240bd9958dc8b72fe90e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:22:52.409402 11701 consensus_queue.cc:1035] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c62adc3b00c449baa2d19f484f6278bf" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:52.409554 11697 consensus_queue.cc:1035] T d5c992efa96140d5a937d7ede28f42bc P 4dde00539a1240bd9958dc8b72fe90e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e82a2d3665f4c1b92efa804edc060ba" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 34717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:52.412729 11706 mvcc.cc:205] Tried to move back new op lower bound from 7317686568584445952 to 7317686568516435968. Current Snapshot: MvccSnapshot[applied={T|T < 7317686568584445952}]
I20260812 14:22:52.413326 11708 mvcc.cc:205] Tried to move back new op lower bound from 7317686568584445952 to 7317686568516435968. Current Snapshot: MvccSnapshot[applied={T|T < 7317686568584445952}]
I20260812 14:22:52.415174 11141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:52.415257 11141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:22:52.416105 11141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa052ca123f042f3aceaebede810040b: Sending DeleteTablet for 3 replicas of tablet d5c992efa96140d5a937d7ede28f42bc
I20260812 14:22:52.416523 11282 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c992efa96140d5a937d7ede28f42bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:52 UTC) from {username='slave'} at 127.0.0.1:45664
I20260812 14:22:52.416597 11436 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c992efa96140d5a937d7ede28f42bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:52 UTC) from {username='slave'} at 127.0.0.1:54322
I20260812 14:22:52.416590 11592 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c992efa96140d5a937d7ede28f42bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:22:52 UTC) from {username='slave'} at 127.0.0.1:45076
I20260812 14:22:52.416887 11718 tablet_replica.cc:330] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba: stopping tablet replica
I20260812 14:22:52.417002 11718 raft_consensus.cc:2229] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:52.417086 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 11212
I20260812 14:22:52.417114 11718 raft_consensus.cc:2258] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:52.418107 11718 ts_tablet_manager.cc:1905] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:52.418741 11719 tablet_replica.cc:330] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf: stopping tablet replica
I20260812 14:22:52.418872 11719 raft_consensus.cc:2229] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:52.418993 11719 raft_consensus.cc:2258] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:52.419543 11719 ts_tablet_manager.cc:1905] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:52.420506 11718 ts_tablet_manager.cc:1918] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:22:52.420605 11718 log.cc:1196] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/d5c992efa96140d5a937d7ede28f42bc
I20260812 14:22:52.420866 11718 ts_tablet_manager.cc:1939] T d5c992efa96140d5a937d7ede28f42bc P 1e82a2d3665f4c1b92efa804edc060ba: Deleting consensus metadata
I20260812 14:22:52.421365 11127 catalog_manager.cc:4841] TS 1e82a2d3665f4c1b92efa804edc060ba (127.5.148.2:34717): tablet d5c992efa96140d5a937d7ede28f42bc (table test-table [id=bff85cb9ce6842acb8e87363104cceb1]) successfully deleted
I20260812 14:22:52.421917 11719 ts_tablet_manager.cc:1918] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:22:52.422022 11719 log.cc:1196] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/d5c992efa96140d5a937d7ede28f42bc
I20260812 14:22:52.422364 11719 ts_tablet_manager.cc:1939] T d5c992efa96140d5a937d7ede28f42bc P c62adc3b00c449baa2d19f484f6278bf: Deleting consensus metadata
I20260812 14:22:52.422864 11127 catalog_manager.cc:4841] TS c62adc3b00c449baa2d19f484f6278bf (127.5.148.3:40329): tablet d5c992efa96140d5a937d7ede28f42bc (table test-table [id=bff85cb9ce6842acb8e87363104cceb1]) successfully deleted
W20260812 14:22:52.429296 11126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:45549 (error 108)
I20260812 14:22:52.429553 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 11363
I20260812 14:22:52.436244 11685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.436025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:45549 (local address 127.0.0.1:45686)
0812 14:22:52.436105 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:52.436192 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:45549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 14:22:52.436431 11126 catalog_manager.cc:4584] TS 4dde00539a1240bd9958dc8b72fe90e6 (127.5.148.1:45549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d5c992efa96140d5a937d7ede28f42bc: Network error: Client connection negotiation failed: client connection to 127.5.148.1:45549: connect: Connection refused (error 111)
I20260812 14:22:52.440621 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 11519
I20260812 14:22:52.451309 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 11107
2026-08-12T14:22:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (697 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 14:22:52.476213 5712 mini_oidc.cc:152] Starting JWKS server
I20260812 14:22:52.477603 5712 webserver.cc:460] Webserver started at https://127.0.0.1:42439/ using document root <none> and password file <none>
I20260812 14:22:52.477697 5712 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 14:22:52.477926 5712 webserver.cc:460] Webserver started at http://127.0.0.1:34337/ using document root <none> and password file <none>
2026-08-12T14:22:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:52Z Disabled control of system clock
I20260812 14:22:52.488943 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:37611
--webserver_interface=127.5.148.62
--webserver_port=0
--builtin_ntp_servers=127.5.148.20:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:37611
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42439/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/testchainca.cert with env {}
W20260812 14:22:52.577587 11726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:52.577847 11726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:52.577875 11726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:52.579064 11726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:52.579111 11726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:52.579126 11726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:52.579141 11726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:52.579155 11726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 14:22:52.579170 11726 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 14:22:52.579183 11726 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42439/jwks.json
I20260812 14:22:52.580832 11726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:37611
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:37611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42439/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:52.581193 11726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:52.581458 11726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:52.586815 11733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:52.586840 11732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:52.586865 11726 server_base.cc:1034] running on GCE node
W20260812 14:22:52.586840 11736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:52.587368 11726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:52.587714 11726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:52.588923 11726 hybrid_clock.cc:648] HybridClock initialized: now 1786544572588903 us; error 30 us; skew 500 ppm
I20260812 14:22:52.590569 11726 webserver.cc:460] Webserver started at http://127.5.148.62:41451/ using document root <none> and password file <none>
I20260812 14:22:52.590844 11726 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:52.590926 11726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:52.591089 11726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:52.592346 11726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "86357b0edad14301a0bea1ec4d88ad7f"
format_stamp: "Formatted at 2026-08-12 14:22:52 on dist-test-slave-8w9v"
server_key: "fc1a51eed7463858cd4d6b8a44de6dfa"
server_key_iv: "aa27faa96114371de37df076db7398a8"
server_key_version: "encryptionkey@0"
I20260812 14:22:52.592756 11726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "86357b0edad14301a0bea1ec4d88ad7f"
format_stamp: "Formatted at 2026-08-12 14:22:52 on dist-test-slave-8w9v"
server_key: "fc1a51eed7463858cd4d6b8a44de6dfa"
server_key_iv: "aa27faa96114371de37df076db7398a8"
server_key_version: "encryptionkey@0"
I20260812 14:22:52.594396 11726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:22:52.595289 11744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:52.595525 11726 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:52.595649 11726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "86357b0edad14301a0bea1ec4d88ad7f"
format_stamp: "Formatted at 2026-08-12 14:22:52 on dist-test-slave-8w9v"
server_key: "fc1a51eed7463858cd4d6b8a44de6dfa"
server_key_iv: "aa27faa96114371de37df076db7398a8"
server_key_version: "encryptionkey@0"
I20260812 14:22:52.595736 11726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:52.622563 11726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:52.622983 11726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:52.623172 11726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:52.627820 11726 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:37611
I20260812 14:22:52.627859 11807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:37611 every 8 connection(s)
I20260812 14:22:52.628280 11726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:52.629460 11808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:52.631879 11808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f: Bootstrap starting.
I20260812 14:22:52.632572 11808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:52.632835 11808 log.cc:824] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:52.633811 11808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f: No bootstrap required, opened a new log
I20260812 14:22:52.634893 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 11726
I20260812 14:22:52.635015 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:52.635229 5712 external_mini_cluster.cc:1428] Setting key d6307bc4fd6c1272e76741a06ef447d0
I20260812 14:22:52.636310 11808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86357b0edad14301a0bea1ec4d88ad7f" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37611 } }
I20260812 14:22:52.636502 11808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:52.636548 11808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86357b0edad14301a0bea1ec4d88ad7f, State: Initialized, Role: FOLLOWER
I20260812 14:22:52.636700 11808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86357b0edad14301a0bea1ec4d88ad7f" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37611 } }
I20260812 14:22:52.636806 11808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:52.636833 11808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:52.636914 11808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:52.637547 11808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86357b0edad14301a0bea1ec4d88ad7f" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37611 } }
I20260812 14:22:52.637645 11808 leader_election.cc:302] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86357b0edad14301a0bea1ec4d88ad7f; no voters:
I20260812 14:22:52.637843 11808 leader_election.cc:288] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:52.637905 11814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:52.638054 11814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [term 1 LEADER]: Becoming Leader. State: Replica: 86357b0edad14301a0bea1ec4d88ad7f, State: Running, Role: LEADER
I20260812 14:22:52.638152 11808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:52.638144 11814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86357b0edad14301a0bea1ec4d88ad7f" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37611 } }
I20260812 14:22:52.638526 11815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86357b0edad14301a0bea1ec4d88ad7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86357b0edad14301a0bea1ec4d88ad7f" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37611 } } }
I20260812 14:22:52.638619 11815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:52.638911 11817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86357b0edad14301a0bea1ec4d88ad7f. Latest consensus state: current_term: 1 leader_uuid: "86357b0edad14301a0bea1ec4d88ad7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86357b0edad14301a0bea1ec4d88ad7f" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37611 } } }
I20260812 14:22:52.639056 11817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:52.639433 11820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:22:52.640082 11820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:22:52.642061 11820 catalog_manager.cc:1347] Generated new cluster ID: a00ed966d4664cb8ac0c7e24dc03ccf2
I20260812 14:22:52.642123 11820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:22:52.642445 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.635825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45826 (local address 127.5.148.62:37611)
0812 14:22:52.636049 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:52.636056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:52.636082 (+ 26us) server_negotiation.cc:407] Connection header received
0812 14:22:52.636271 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:52.636277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:52.636337 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:52.636455 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:52.636774 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:52.638475 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:52.639704 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:52.640416 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:52.640552 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:52.641593 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:52.641597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:52.641597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:52.641785 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:52.641820 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:52.641831 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:52.641908 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:52.642136 (+ 228us) server_negotiation.cc:299] Negotiation successful
0812 14:22:52.642222 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260812 14:22:52.643514 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:22:52.657612 11820 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:52.658141 11820 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:52.667088 11820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86357b0edad14301a0bea1ec4d88ad7f: Generated new TSK 0
I20260812 14:22:52.667309 11820 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 14:22:52.696278 11841 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 14:22:52.696483 11841 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 14:22:52.696790 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.675588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45834 (local address 127.5.148.62:37611)
0812 14:22:52.675661 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:52.675663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:52.675905 (+ 242us) server_negotiation.cc:407] Connection header received
0812 14:22:52.675948 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:52.675949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:52.675961 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:52.676049 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0812 14:22:52.676271 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:52.676776 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:52.677959 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:52.678201 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:52.678265 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:52.678500 (+ 235us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 14:22:52.696452 (+ 17952us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 14:22:52.696731 (+ 279us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45834: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260812 14:22:52.696879 11813 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45834: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 14:22:52.697292 11840 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:22:52.675446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37611 (local address 127.0.0.1:45834)
0812 14:22:52.675673 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:52.675687 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:52.675784 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:52.676084 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:52.676085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:52.676088 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0812 14:22:52.676189 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:52.676192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:52.676860 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:52.676861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:52.677828 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:52.677832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:52.678285 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:52.678287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:52.678358 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:52.678385 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 14:22:52.696689 (+ 18304us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 14:22:52.696851 (+ 162us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.148.62:37611: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":108,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.148.62:37611: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 14:22:52.697808 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 11726
2026-08-12T14:22:52Z chronyd exiting
I20260812 14:22:52.737897 5712 test_util.cc:175] -----------------------------------------------
I20260812 14:22:52.737990 5712 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786544547303129-5712-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (263 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 14:22:52.738919 5712 mini_oidc.cc:152] Starting JWKS server
I20260812 14:22:52.739665 5712 webserver.cc:460] Webserver started at https://127.0.0.1:40991/ using document root <none> and password file <none>
I20260812 14:22:52.739710 5712 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 14:22:52.739904 5712 webserver.cc:460] Webserver started at http://127.0.0.1:42587/ using document root <none> and password file <none>
2026-08-12T14:22:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:52Z Disabled control of system clock
I20260812 14:22:52.749816 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:36093
--webserver_interface=127.5.148.62
--webserver_port=0
--builtin_ntp_servers=127.5.148.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:36093
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40991/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/testchainca.cert with env {}
W20260812 14:22:52.834924 11847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:52.835144 11847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:52.835166 11847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:52.836321 11847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:52.836360 11847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:52.836378 11847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:52.836390 11847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:52.836403 11847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 14:22:52.836416 11847 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 14:22:52.836426 11847 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40991/jwks.json
I20260812 14:22:52.837888 11847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:36093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:36093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40991/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:52.838104 11847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:52.838274 11847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:52.843248 11852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:52.843236 11853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:52.843252 11855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:52.843497 11847 server_base.cc:1034] running on GCE node
I20260812 14:22:52.843739 11847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:52.844029 11847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:52.845184 11847 hybrid_clock.cc:648] HybridClock initialized: now 1786544572845168 us; error 36 us; skew 500 ppm
I20260812 14:22:52.846521 11847 webserver.cc:460] Webserver started at http://127.5.148.62:33353/ using document root <none> and password file <none>
I20260812 14:22:52.846791 11847 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:52.846837 11847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:52.847028 11847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:52.848112 11847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "eb8e77757db44134a3aeb1838deed6c3"
format_stamp: "Formatted at 2026-08-12 14:22:52 on dist-test-slave-8w9v"
server_key: "826a93aaf0de58c48ec77288205a22cc"
server_key_iv: "a6369f344a8e740a573719396f1922e5"
server_key_version: "encryptionkey@0"
I20260812 14:22:52.848567 11847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "eb8e77757db44134a3aeb1838deed6c3"
format_stamp: "Formatted at 2026-08-12 14:22:52 on dist-test-slave-8w9v"
server_key: "826a93aaf0de58c48ec77288205a22cc"
server_key_iv: "a6369f344a8e740a573719396f1922e5"
server_key_version: "encryptionkey@0"
I20260812 14:22:52.849946 11847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:22:52.850769 11861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:52.850999 11847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:52.851109 11847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "eb8e77757db44134a3aeb1838deed6c3"
format_stamp: "Formatted at 2026-08-12 14:22:52 on dist-test-slave-8w9v"
server_key: "826a93aaf0de58c48ec77288205a22cc"
server_key_iv: "a6369f344a8e740a573719396f1922e5"
server_key_version: "encryptionkey@0"
I20260812 14:22:52.851199 11847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:52.865875 11847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:52.866250 11847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:52.866420 11847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:52.870836 11847 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:36093
I20260812 14:22:52.870877 11923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:36093 every 8 connection(s)
I20260812 14:22:52.871271 11847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:52.872411 11925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:52.874822 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 11847
I20260812 14:22:52.874881 11925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3: Bootstrap starting.
I20260812 14:22:52.874963 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:52.875155 5712 external_mini_cluster.cc:1428] Setting key a840b980daf472eea4ed58a20a7008e6
I20260812 14:22:52.875592 11925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:52.875941 11925 log.cc:824] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:52.876783 11925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3: No bootstrap required, opened a new log
I20260812 14:22:52.878911 11925 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8e77757db44134a3aeb1838deed6c3" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 36093 } }
I20260812 14:22:52.879105 11925 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:52.879136 11925 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb8e77757db44134a3aeb1838deed6c3, State: Initialized, Role: FOLLOWER
I20260812 14:22:52.879316 11925 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8e77757db44134a3aeb1838deed6c3" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 36093 } }
I20260812 14:22:52.879398 11925 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:52.879449 11925 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:52.879499 11925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:52.880365 11925 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8e77757db44134a3aeb1838deed6c3" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 36093 } }
I20260812 14:22:52.880496 11925 leader_election.cc:302] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb8e77757db44134a3aeb1838deed6c3; no voters:
I20260812 14:22:52.880702 11925 leader_election.cc:288] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:52.880728 11930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:52.880842 11930 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [term 1 LEADER]: Becoming Leader. State: Replica: eb8e77757db44134a3aeb1838deed6c3, State: Running, Role: LEADER
I20260812 14:22:52.880937 11930 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8e77757db44134a3aeb1838deed6c3" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 36093 } }
I20260812 14:22:52.880993 11925 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:52.881271 11932 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb8e77757db44134a3aeb1838deed6c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8e77757db44134a3aeb1838deed6c3" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 36093 } } }
I20260812 14:22:52.881405 11932 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:52.881505 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.875768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.5.148.62:36093)
0812 14:22:52.875979 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:52.875997 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:52.876023 (+ 26us) server_negotiation.cc:407] Connection header received
0812 14:22:52.876547 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:52.876554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:52.876622 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:52.876746 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:52.877001 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:52.877888 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:52.879053 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:52.879658 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:52.879751 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:52.880528 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:52.880533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:52.880534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:52.880734 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:52.880801 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:52.880817 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:52.881164 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:52.881198 (+ 34us) server_negotiation.cc:299] Negotiation successful
0812 14:22:52.881285 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260812 14:22:52.881968 11931 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb8e77757db44134a3aeb1838deed6c3. Latest consensus state: current_term: 1 leader_uuid: "eb8e77757db44134a3aeb1838deed6c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8e77757db44134a3aeb1838deed6c3" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 36093 } } }
I20260812 14:22:52.882099 11931 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:52.882619 11936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:22:52.883369 11936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:22:52.883752 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:22:52.885151 11936 catalog_manager.cc:1347] Generated new cluster ID: 5b7b2d4e86c9423e8ea6e200acbff0dd
I20260812 14:22:52.885254 11936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:22:52.912817 11936 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:52.913455 11936 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:52.916615 11936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb8e77757db44134a3aeb1838deed6c3: Generated new TSK 0
I20260812 14:22:52.916847 11936 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 14:22:52.940564 11962 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 14:22:52.940663 11962 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 14:22:52.941017 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:52.926457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58288 (local address 127.5.148.62:36093)
0812 14:22:52.926528 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:52.926530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:52.926680 (+ 150us) server_negotiation.cc:407] Connection header received
0812 14:22:52.926763 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:52.926763 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:52.926775 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:52.926839 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0812 14:22:52.926999 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:52.927485 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:52.928663 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:52.928870 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:52.928938 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:52.929119 (+ 181us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 14:22:52.940816 (+ 11697us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40991/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 14:22:52.940963 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58288: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40991/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":41}
W20260812 14:22:52.941097 11928 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58288: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40991/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 14:22:52.941211 11961 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:22:52.926409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:36093 (local address 127.0.0.1:58288)
0812 14:22:52.926603 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:52.926616 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:52.926689 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:52.926862 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:52.926863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:52.926865 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0812 14:22:52.926934 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:52.926936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:52.927583 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:52.927584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:52.928582 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:52.928585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:52.928952 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:52.928953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:52.929015 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:52.929022 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 14:22:52.940950 (+ 11928us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40991/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 14:22:52.941101 (+ 151us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.148.62:36093: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40991/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":112,"threads_started":1}
I20260812 14:22:52.941565 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 11847
2026-08-12T14:22:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (245 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 14:22:52.984261 5712 mini_oidc.cc:152] Starting JWKS server
I20260812 14:22:52.984993 5712 webserver.cc:460] Webserver started at https://127.0.0.1:35735/ using document root <none> and password file <none>
I20260812 14:22:52.985064 5712 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 14:22:52.985231 5712 webserver.cc:460] Webserver started at http://127.0.0.1:37637/ using document root <none> and password file <none>
2026-08-12T14:22:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:52Z Disabled control of system clock
I20260812 14:22:52.994774 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:39745
--webserver_interface=127.5.148.62
--webserver_port=0
--builtin_ntp_servers=127.5.148.20:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:39745
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35735/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 14:22:53.080268 11968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:53.080494 11968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:53.080518 11968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:53.081744 11968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:53.081791 11968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:53.081804 11968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:53.081817 11968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:53.081830 11968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 14:22:53.081843 11968 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 14:22:53.081857 11968 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35735/jwks.json
I20260812 14:22:53.083389 11968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:39745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:39745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35735/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:53.083622 11968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:53.083878 11968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:53.088981 11976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:53.088987 11973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:53.089197 11974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:53.089095 11968 server_base.cc:1034] running on GCE node
I20260812 14:22:53.089533 11968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:53.089902 11968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:53.091058 11968 hybrid_clock.cc:648] HybridClock initialized: now 1786544573091032 us; error 37 us; skew 500 ppm
I20260812 14:22:53.092442 11968 webserver.cc:460] Webserver started at http://127.5.148.62:41675/ using document root <none> and password file <none>
I20260812 14:22:53.092692 11968 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:53.092741 11968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:53.092917 11968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:53.094089 11968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "dfa504c1455e4cbfaee2fcdce8c7cc58"
format_stamp: "Formatted at 2026-08-12 14:22:53 on dist-test-slave-8w9v"
server_key: "0e7e1dc6bf4822b80cc56ff218bf3b54"
server_key_iv: "6a90be369412c5ad32e99d11a1e3eea5"
server_key_version: "encryptionkey@0"
I20260812 14:22:53.094485 11968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "dfa504c1455e4cbfaee2fcdce8c7cc58"
format_stamp: "Formatted at 2026-08-12 14:22:53 on dist-test-slave-8w9v"
server_key: "0e7e1dc6bf4822b80cc56ff218bf3b54"
server_key_iv: "6a90be369412c5ad32e99d11a1e3eea5"
server_key_version: "encryptionkey@0"
I20260812 14:22:53.095872 11968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:22:53.096732 11983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:53.096894 11968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:53.096997 11968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "dfa504c1455e4cbfaee2fcdce8c7cc58"
format_stamp: "Formatted at 2026-08-12 14:22:53 on dist-test-slave-8w9v"
server_key: "0e7e1dc6bf4822b80cc56ff218bf3b54"
server_key_iv: "6a90be369412c5ad32e99d11a1e3eea5"
server_key_version: "encryptionkey@0"
I20260812 14:22:53.097059 11968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:53.113509 11968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:53.113816 11968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:53.113940 11968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:53.118062 11968 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:39745
I20260812 14:22:53.118099 12042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:39745 every 8 connection(s)
I20260812 14:22:53.118503 11968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:53.119251 12043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:53.119812 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 11968
I20260812 14:22:53.119895 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:53.120091 5712 external_mini_cluster.cc:1428] Setting key 245437ec9562089226ef45d83295117e
I20260812 14:22:53.123590 12043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58: Bootstrap starting.
I20260812 14:22:53.124300 12043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:53.124640 12043 log.cc:824] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:53.125561 12043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58: No bootstrap required, opened a new log
I20260812 14:22:53.126708 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:53.120751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.5.148.62:39745)
0812 14:22:53.120997 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:53.121007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:53.121045 (+ 38us) server_negotiation.cc:407] Connection header received
0812 14:22:53.121648 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:53.121658 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:53.121760 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:53.121924 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:53.122127 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:53.123491 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:53.124357 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:53.124972 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:53.125073 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:53.125960 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:53.125965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:53.125965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:53.126175 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:53.126221 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:53.126236 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:53.126308 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:53.126398 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 14:22:53.126489 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
I20260812 14:22:53.127877 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:22:53.128085 12043 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa504c1455e4cbfaee2fcdce8c7cc58" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39745 } }
I20260812 14:22:53.128283 12043 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:53.128328 12043 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa504c1455e4cbfaee2fcdce8c7cc58, State: Initialized, Role: FOLLOWER
I20260812 14:22:53.128454 12043 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa504c1455e4cbfaee2fcdce8c7cc58" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39745 } }
I20260812 14:22:53.128535 12043 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:53.128556 12043 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:53.128621 12043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:53.129295 12043 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa504c1455e4cbfaee2fcdce8c7cc58" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39745 } }
I20260812 14:22:53.129419 12043 leader_election.cc:302] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfa504c1455e4cbfaee2fcdce8c7cc58; no voters:
I20260812 14:22:53.129598 12043 leader_election.cc:288] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:53.129629 12049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:53.129782 12049 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [term 1 LEADER]: Becoming Leader. State: Replica: dfa504c1455e4cbfaee2fcdce8c7cc58, State: Running, Role: LEADER
I20260812 14:22:53.129877 12049 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa504c1455e4cbfaee2fcdce8c7cc58" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39745 } }
I20260812 14:22:53.129884 12043 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:53.130275 12050 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfa504c1455e4cbfaee2fcdce8c7cc58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa504c1455e4cbfaee2fcdce8c7cc58" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39745 } } }
I20260812 14:22:53.130275 12051 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfa504c1455e4cbfaee2fcdce8c7cc58. Latest consensus state: current_term: 1 leader_uuid: "dfa504c1455e4cbfaee2fcdce8c7cc58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa504c1455e4cbfaee2fcdce8c7cc58" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39745 } } }
I20260812 14:22:53.130388 12050 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:53.130393 12051 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:53.130662 12058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:22:53.131165 12058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:22:53.132817 12058 catalog_manager.cc:1347] Generated new cluster ID: d49839f097b54fd297b45dd8cde914fc
I20260812 14:22:53.132875 12058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:22:53.146363 12058 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:53.147140 12058 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:53.158720 12058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfa504c1455e4cbfaee2fcdce8c7cc58: Generated new TSK 0
I20260812 14:22:53.158964 12058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:22:53.175601 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:53.171093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43762 (local address 127.5.148.62:39745)
0812 14:22:53.171167 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:53.171170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:53.171315 (+ 145us) server_negotiation.cc:407] Connection header received
0812 14:22:53.171406 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:53.171407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:53.171420 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:53.171490 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0812 14:22:53.171651 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:53.172244 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:53.173392 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:53.173586 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:53.173637 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:53.173856 (+ 219us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 14:22:53.175347 (+ 1491us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35735/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 14:22:53.175518 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35735/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":43}
W20260812 14:22:53.175668 12078 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:22:53.171039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:39745 (local address 127.0.0.1:43762)
0812 14:22:53.171231 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:53.171243 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:53.171324 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:53.171509 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:53.171510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:53.171512 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0812 14:22:53.171578 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:53.171581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:53.172328 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:53.172329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:53.173303 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:53.173307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:53.173668 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:53.173669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:53.173740 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:53.173747 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 14:22:53.175471 (+ 1724us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35735/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 14:22:53.175570 (+ 99us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.148.62:39745: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35735/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":121,"threads_started":1}
W20260812 14:22:53.175676 12046 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35735/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260812 14:22:53.176355 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 11968
E20260812 14:22:53.191502 12079 webserver.cc:533] Webserver: sslize failed:
E20260812 14:22:53.191596 12079 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-12T14:22:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (235 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T14:22:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:53Z Disabled control of system clock
I20260812 14:22:53.229301 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:39141
--webserver_interface=127.5.148.62
--webserver_port=0
--builtin_ntp_servers=127.5.148.20:43429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:39141
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/testchainca.cert with env {}
W20260812 14:22:53.314829 12083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:53.315052 12083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:53.315076 12083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:53.316354 12083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:53.316406 12083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:53.316443 12083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:53.316457 12083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:53.316469 12083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 14:22:53.316529 12083 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 14:22:53.316547 12083 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 14:22:53.318127 12083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:43429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:39141
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:39141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:53.318423 12083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:53.318655 12083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:53.323956 12088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:53.324038 12093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:53.324186 12083 server_base.cc:1034] running on GCE node
W20260812 14:22:53.323956 12091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:53.324563 12083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:53.324956 12083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:53.326154 12083 hybrid_clock.cc:648] HybridClock initialized: now 1786544573326130 us; error 53 us; skew 500 ppm
I20260812 14:22:53.327493 12083 webserver.cc:460] Webserver started at http://127.5.148.62:43009/ using document root <none> and password file <none>
I20260812 14:22:53.327771 12083 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:53.327819 12083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:53.327962 12083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:53.329098 12083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "7bee84f2a0374d7ea162c96126d91036"
format_stamp: "Formatted at 2026-08-12 14:22:53 on dist-test-slave-8w9v"
server_key: "d1aa62c09fe7e428af4aa94bdbc690ae"
server_key_iv: "9b61d947a695f389921451c496de9ec7"
server_key_version: "encryptionkey@0"
I20260812 14:22:53.329478 12083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "7bee84f2a0374d7ea162c96126d91036"
format_stamp: "Formatted at 2026-08-12 14:22:53 on dist-test-slave-8w9v"
server_key: "d1aa62c09fe7e428af4aa94bdbc690ae"
server_key_iv: "9b61d947a695f389921451c496de9ec7"
server_key_version: "encryptionkey@0"
I20260812 14:22:53.330839 12083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:22:53.331626 12100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:53.331811 12083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:22:53.331907 12083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "7bee84f2a0374d7ea162c96126d91036"
format_stamp: "Formatted at 2026-08-12 14:22:53 on dist-test-slave-8w9v"
server_key: "d1aa62c09fe7e428af4aa94bdbc690ae"
server_key_iv: "9b61d947a695f389921451c496de9ec7"
server_key_version: "encryptionkey@0"
I20260812 14:22:53.332010 12083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:53.348842 12083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:53.349200 12083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:53.349357 12083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:53.353598 12083 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:39141
I20260812 14:22:53.353667 12158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:39141 every 8 connection(s)
I20260812 14:22:53.354046 12083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:53.354612 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 12083
I20260812 14:22:53.354759 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:53.355014 5712 external_mini_cluster.cc:1428] Setting key fb8048eab5cdce0285608361f1ecba84
I20260812 14:22:53.355484 12159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:53.359007 12159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036: Bootstrap starting.
I20260812 14:22:53.359735 12159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:53.360107 12159 log.cc:824] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:53.360944 12163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:53.355648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38116 (local address 127.5.148.62:39141)
0812 14:22:53.355862 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:53.355870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:53.355895 (+ 25us) server_negotiation.cc:407] Connection header received
0812 14:22:53.356429 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:53.356436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:53.356501 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:53.356622 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:53.356917 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:53.357804 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:53.358904 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:53.359359 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:53.359458 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:53.360265 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:53.360269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:53.360270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:22:53.360469 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:53.360503 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:22:53.360514 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:53.360601 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:53.360684 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 14:22:53.360772 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260812 14:22:53.360966 12159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036: No bootstrap required, opened a new log
I20260812 14:22:53.362092 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:22:53.363265 12159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bee84f2a0374d7ea162c96126d91036" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39141 } }
I20260812 14:22:53.363408 12159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:53.363431 12159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bee84f2a0374d7ea162c96126d91036, State: Initialized, Role: FOLLOWER
I20260812 14:22:53.363580 12159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bee84f2a0374d7ea162c96126d91036" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39141 } }
I20260812 14:22:53.363646 12159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:53.363669 12159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:53.363735 12159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:53.364413 12159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bee84f2a0374d7ea162c96126d91036" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39141 } }
I20260812 14:22:53.364531 12159 leader_election.cc:302] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bee84f2a0374d7ea162c96126d91036; no voters:
I20260812 14:22:53.364717 12159 leader_election.cc:288] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:53.364804 12165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:53.364987 12159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:53.364985 12165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [term 1 LEADER]: Becoming Leader. State: Replica: 7bee84f2a0374d7ea162c96126d91036, State: Running, Role: LEADER
I20260812 14:22:53.365121 12165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bee84f2a0374d7ea162c96126d91036" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39141 } }
I20260812 14:22:53.365517 12166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bee84f2a0374d7ea162c96126d91036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bee84f2a0374d7ea162c96126d91036" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39141 } } }
I20260812 14:22:53.365504 12167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bee84f2a0374d7ea162c96126d91036. Latest consensus state: current_term: 1 leader_uuid: "7bee84f2a0374d7ea162c96126d91036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bee84f2a0374d7ea162c96126d91036" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39141 } } }
I20260812 14:22:53.365615 12166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:53.365620 12167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:53.365859 12175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:22:53.366460 12175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:22:53.372551 12175 catalog_manager.cc:1347] Generated new cluster ID: a33bc7586d564ddfaeef698464a77138
I20260812 14:22:53.372617 12175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:22:53.392827 12175 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:53.393607 12175 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:53.403923 12175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bee84f2a0374d7ea162c96126d91036: Generated new TSK 0
I20260812 14:22:53.404230 12175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:22:53.485440 12163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:53.418706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.5.148.62:39141)
0812 14:22:53.418773 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:53.418775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:53.418886 (+ 111us) server_negotiation.cc:407] Connection header received
0812 14:22:53.418942 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:53.418943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:53.418958 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:53.419056 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0812 14:22:53.419177 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:53.419971 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:53.420852 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:53.421173 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:53.421253 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:53.421442 (+ 189us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 14:22:53.485117 (+ 63675us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 14:22:53.485356 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39124: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":39}
W20260812 14:22:53.485529 12163 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39124: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 14:22:53.485505 12193 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:22:53.418619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:39141 (local address 127.0.0.1:39124)
0812 14:22:53.418812 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:53.418823 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:53.418883 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:53.419060 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:53.419061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:53.419063 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0812 14:22:53.419117 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:53.419119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:53.420097 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:53.420098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:53.420748 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:53.420751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:53.421267 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:53.421269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:53.421317 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:53.421324 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 14:22:53.485283 (+ 63959us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 14:22:53.485426 (+ 143us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.148.62:39141: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":116,"threads_started":1}
I20260812 14:22:53.486052 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 12083
2026-08-12T14:22:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (288 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (98 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (95 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:22:53 dist-test-slave-8w9v krb5kdc[12207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:22:53 dist-test-slave-8w9v krb5kdc[12207](info): set up 2 sockets
Aug 12 14:22:53 dist-test-slave-8w9v krb5kdc[12207](info): commencing operation
krb5kdc: starting...
W20260812 14:22:54.822444 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:22:54 dist-test-slave-8w9v krb5kdc[12207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:22:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:22:54.955920 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:34487
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:41359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:34487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:55.042143 12225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:55.042414 12225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:55.042454 12225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:55.043689 12225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:55.043771 12225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:55.043795 12225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:55.043816 12225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:55.043848 12225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:22:55.045428 12225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:34487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:34487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:55.045719 12225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:55.045976 12225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:55.051236 12230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:55.051303 12234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:55.051236 12231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:55.051407 12225 server_base.cc:1034] running on GCE node
I20260812 14:22:55.051757 12225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:55.052038 12225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:55.053248 12225 hybrid_clock.cc:648] HybridClock initialized: now 1786544575053239 us; error 33 us; skew 500 ppm
Aug 12 14:22:55 dist-test-slave-8w9v krb5kdc[12207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544575, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:55.064831 12225 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:22:55.065470 12225 webserver.cc:460] Webserver started at http://127.5.148.62:43611/ using document root <none> and password file <none>
I20260812 14:22:55.065786 12225 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:55.065846 12225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:55.066027 12225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:55.067281 12225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "cb8be8b27f4845edb965dedfcc09bb59"
format_stamp: "Formatted at 2026-08-12 14:22:55 on dist-test-slave-8w9v"
server_key: "a644a8f1072eca51930256e5d564589d"
server_key_iv: "c738d54ffee6dc2f306de6449fab3ab7"
server_key_version: "encryptionkey@0"
I20260812 14:22:55.067715 12225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "cb8be8b27f4845edb965dedfcc09bb59"
format_stamp: "Formatted at 2026-08-12 14:22:55 on dist-test-slave-8w9v"
server_key: "a644a8f1072eca51930256e5d564589d"
server_key_iv: "c738d54ffee6dc2f306de6449fab3ab7"
server_key_version: "encryptionkey@0"
I20260812 14:22:55.069432 12225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260812 14:22:55.070340 12244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:55.070567 12225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:55.070664 12225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "cb8be8b27f4845edb965dedfcc09bb59"
format_stamp: "Formatted at 2026-08-12 14:22:55 on dist-test-slave-8w9v"
server_key: "a644a8f1072eca51930256e5d564589d"
server_key_iv: "c738d54ffee6dc2f306de6449fab3ab7"
server_key_version: "encryptionkey@0"
I20260812 14:22:55.070730 12225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:55.081296 12225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:55.083523 12225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:55.083655 12225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:55.088094 12225 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:34487
I20260812 14:22:55.088198 12307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:34487 every 8 connection(s)
I20260812 14:22:55.088596 12225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:55.089550 12308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:55.091984 12308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59: Bootstrap starting.
I20260812 14:22:55.092360 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 12225
I20260812 14:22:55.092473 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:55.092667 5712 external_mini_cluster.cc:1428] Setting key 8c6e82db2d04e07bb9287ccfff4e72b7
I20260812 14:22:55.092849 12308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:55.093233 12308 log.cc:824] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:55.094117 12308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59: No bootstrap required, opened a new log
I20260812 14:22:55.096655 12308 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8be8b27f4845edb965dedfcc09bb59" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 34487 } }
I20260812 14:22:55.096815 12308 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:55.096840 12308 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8be8b27f4845edb965dedfcc09bb59, State: Initialized, Role: FOLLOWER
I20260812 14:22:55.096990 12308 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8be8b27f4845edb965dedfcc09bb59" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 34487 } }
I20260812 14:22:55.097057 12308 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:55.097079 12308 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:55.097148 12308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:55.097915 12308 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8be8b27f4845edb965dedfcc09bb59" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 34487 } }
I20260812 14:22:55.098038 12308 leader_election.cc:302] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb8be8b27f4845edb965dedfcc09bb59; no voters:
I20260812 14:22:55.098227 12308 leader_election.cc:288] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:55.098284 12313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:55.098495 12313 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [term 1 LEADER]: Becoming Leader. State: Replica: cb8be8b27f4845edb965dedfcc09bb59, State: Running, Role: LEADER
I20260812 14:22:55.098651 12313 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8be8b27f4845edb965dedfcc09bb59" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 34487 } }
I20260812 14:22:55.098536 12308 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:55.099004 12313 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb8be8b27f4845edb965dedfcc09bb59. Latest consensus state: current_term: 1 leader_uuid: "cb8be8b27f4845edb965dedfcc09bb59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8be8b27f4845edb965dedfcc09bb59" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 34487 } } }
I20260812 14:22:55.099016 12315 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb8be8b27f4845edb965dedfcc09bb59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8be8b27f4845edb965dedfcc09bb59" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 34487 } } }
I20260812 14:22:55.099097 12313 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:55.099102 12315 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:55.099380 12317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:22:55.099934 12317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 14:22:55 dist-test-slave-8w9v krb5kdc[12207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:55.101836 12317 catalog_manager.cc:1347] Generated new cluster ID: f84145754f004a348f78f5d04fd2e2a3
I20260812 14:22:55.101883 12317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:22:55.106483 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:55.093248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.5.148.62:34487)
0812 14:22:55.093490 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:55.093500 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:55.093536 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:22:55.094116 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:55.094126 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:55.094190 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:55.094358 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:55.094992 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:55.096533 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:55.097515 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:55.098185 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:55.098662 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:55.101578 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:55.101582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:55.101583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:55.101597 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:55.104016 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:55.104573 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:55.104576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:55.104577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:55.104698 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:55.104986 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:55.104987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:55.104987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:55.105313 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:55.105965 (+ 652us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:55.106061 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 14:22:55.106225 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":140,"threads_started":1}
I20260812 14:22:55.109632 12317 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:55.110170 12317 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:55.115854 12317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb8be8b27f4845edb965dedfcc09bb59: Generated new TSK 0
I20260812 14:22:55.116179 12317 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
W20260812 14:22:55.897439 5712 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.148.1: real 0.779s user 0.000s sys 0.005s
I20260812 14:22:55.897855 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:34487
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:41359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:55.990902 12338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:55.991250 12338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:55.991290 12338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:55.992693 12338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:55.992745 12338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:55.992775 12338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:22:55.994388 12338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:55.994660 12338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:55.994921 12338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:56.000775 12343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:56.000809 12347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:56.000804 12344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:56.000952 12338 server_base.cc:1034] running on GCE node
I20260812 14:22:56.001284 12338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:56.001530 12338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:56.002715 12338 hybrid_clock.cc:648] HybridClock initialized: now 1786544576002701 us; error 31 us; skew 500 ppm
Aug 12 14:22:56 dist-test-slave-8w9v krb5kdc[12207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544576, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:56.005982 12338 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:22:56.006503 12338 webserver.cc:460] Webserver started at http://127.5.148.1:34363/ using document root <none> and password file <none>
I20260812 14:22:56.006770 12338 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:56.006844 12338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:56.006996 12338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:56.008176 12338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "231ff4ad14844c7aa072e5dc87e85c4f"
format_stamp: "Formatted at 2026-08-12 14:22:56 on dist-test-slave-8w9v"
server_key: "3246184c7430f059302149e3c7b6e19b"
server_key_iv: "e782901b01fba2ce2d76553bef1aebce"
server_key_version: "encryptionkey@0"
I20260812 14:22:56.008575 12338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "231ff4ad14844c7aa072e5dc87e85c4f"
format_stamp: "Formatted at 2026-08-12 14:22:56 on dist-test-slave-8w9v"
server_key: "3246184c7430f059302149e3c7b6e19b"
server_key_iv: "e782901b01fba2ce2d76553bef1aebce"
server_key_version: "encryptionkey@0"
I20260812 14:22:56.010056 12338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:22:56.010900 12355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:56.011081 12338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:22:56.011197 12338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "231ff4ad14844c7aa072e5dc87e85c4f"
format_stamp: "Formatted at 2026-08-12 14:22:56 on dist-test-slave-8w9v"
server_key: "3246184c7430f059302149e3c7b6e19b"
server_key_iv: "e782901b01fba2ce2d76553bef1aebce"
server_key_version: "encryptionkey@0"
I20260812 14:22:56.011328 12338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:56.026981 12338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:56.029349 12338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:56.029568 12338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:56.029843 12338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:56.030232 12338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:56.030303 12338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:56.030378 12338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:56.030413 12338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:56.037811 12338 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:33069
I20260812 14:22:56.037860 12491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:33069 every 8 connection(s)
I20260812 14:22:56.038216 12338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:22:56.044932 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 12338
I20260812 14:22:56.045075 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:22:56.045321 5712 external_mini_cluster.cc:1428] Setting key 186c32665e1ada731a0b63c9ed9ccbb1
Aug 12 14:22:56 dist-test-slave-8w9v krb5kdc[12207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544576, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:56.049856 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:56.039330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:59863 (local address 127.5.148.62:34487)
0812 14:22:56.039463 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:56.039465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:56.039570 (+ 105us) server_negotiation.cc:407] Connection header received
0812 14:22:56.040733 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:56.040734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:56.040747 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:56.040831 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:56.041545 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:56.042078 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:56.043295 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:56.043515 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:56.043577 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:56.046382 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:56.046383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:56.046384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:56.046391 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:56.048233 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:56.048822 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:56.048823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:56.048823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:56.048880 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:56.049168 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:56.049168 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:56.049169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:56.049268 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:56.049315 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:56.049697 (+ 382us) server_negotiation.cc:299] Negotiation successful
0812 14:22:56.049781 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260812 14:22:56.050185 12495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:56.039244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:34487 (local address 127.5.148.1:59863)
0812 14:22:56.039425 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:56.039475 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:56.040619 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:56.040928 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:56.040931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:56.041209 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:56.041417 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:56.041422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:56.042169 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:56.042169 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:56.043192 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:56.043195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:56.043600 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:56.043601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:56.043717 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:56.044397 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:56.044429 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:56.046287 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:56.048324 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:56.048331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:56.048333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:56.048745 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:56.048954 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:56.048954 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:56.048955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:56.049103 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:56.049329 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:56.049335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:56.049520 (+ 185us) client_negotiation.cc:769] Sending connection context
0812 14:22:56.049648 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 14:22:56.049889 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260812 14:22:56.050697 12492 heartbeater.cc:347] Connected to a master server at 127.5.148.62:34487
I20260812 14:22:56.050822 12492 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:56.051038 12492 heartbeater.cc:511] Master 127.5.148.62:34487 requested a full tablet report, sending...
I20260812 14:22:56.051602 12266 ts_manager.cc:194] Registered new tserver with Master: 231ff4ad14844c7aa072e5dc87e85c4f (127.5.148.1:33069)
I20260812 14:22:56.052377 12266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:59863
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
W20260812 14:22:56.802449 5712 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.148.2: real 0.756s user 0.002s sys 0.002s
I20260812 14:22:56.802816 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:34487
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:41359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:56.895519 12500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:56.895745 12500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:56.895771 12500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:56.896932 12500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:56.896974 12500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:56.897001 12500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:22:56.898587 12500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:56.898792 12500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:56.899067 12500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:56.904476 12505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:56.904476 12508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:56.904587 12506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:56.904870 12500 server_base.cc:1034] running on GCE node
I20260812 14:22:56.905128 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:56.905397 12500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:56.906571 12500 hybrid_clock.cc:648] HybridClock initialized: now 1786544576906565 us; error 36 us; skew 500 ppm
Aug 12 14:22:56 dist-test-slave-8w9v krb5kdc[12207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544576, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:56.909886 12500 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:22:56.910538 12500 webserver.cc:460] Webserver started at http://127.5.148.2:36965/ using document root <none> and password file <none>
I20260812 14:22:56.910843 12500 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:56.910897 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:56.911006 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:56.912418 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "24fe3a01a31840d2ad33a282ee62d99f"
format_stamp: "Formatted at 2026-08-12 14:22:56 on dist-test-slave-8w9v"
server_key: "a62d4dd45d0f481767ba294edfe7a483"
server_key_iv: "b7538e4a13e25f97568aa6be91048a78"
server_key_version: "encryptionkey@0"
I20260812 14:22:56.912746 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "24fe3a01a31840d2ad33a282ee62d99f"
format_stamp: "Formatted at 2026-08-12 14:22:56 on dist-test-slave-8w9v"
server_key: "a62d4dd45d0f481767ba294edfe7a483"
server_key_iv: "b7538e4a13e25f97568aa6be91048a78"
server_key_version: "encryptionkey@0"
I20260812 14:22:56.914115 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:22:56.915373 12516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:56.915594 12500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:56.915715 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "24fe3a01a31840d2ad33a282ee62d99f"
format_stamp: "Formatted at 2026-08-12 14:22:56 on dist-test-slave-8w9v"
server_key: "a62d4dd45d0f481767ba294edfe7a483"
server_key_iv: "b7538e4a13e25f97568aa6be91048a78"
server_key_version: "encryptionkey@0"
I20260812 14:22:56.915830 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:56.941195 12500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:56.943662 12500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:56.943857 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:56.944191 12500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:56.944584 12500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:56.944645 12500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:56.944682 12500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:56.944710 12500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:56.951713 12500 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:39865
I20260812 14:22:56.951772 12646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:39865 every 8 connection(s)
I20260812 14:22:56.952229 12500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:22:56 dist-test-slave-8w9v krb5kdc[12207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544576, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:56.959517 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 12500
I20260812 14:22:56.959641 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:22:56.959820 5712 external_mini_cluster.cc:1428] Setting key 8c0767fe7725623d4d900364f5cd8ea9
I20260812 14:22:56.962658 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:56.953071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:60089 (local address 127.5.148.62:34487)
0812 14:22:56.953235 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:56.953238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:56.953373 (+ 135us) server_negotiation.cc:407] Connection header received
0812 14:22:56.954346 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:56.954347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:56.954360 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:56.954434 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:56.955114 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:56.955662 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:56.956761 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:56.956996 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:56.957063 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:56.959450 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:56.959451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:56.959451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:56.959458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:56.961120 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:56.961577 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:56.961578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:56.961579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:56.961648 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:56.961929 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:56.961931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:56.961931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:56.962060 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:56.962143 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:56.962457 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 14:22:56.962554 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
I20260812 14:22:56.962870 12652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:56.953019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:34487 (local address 127.5.148.2:60089)
0812 14:22:56.953246 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:56.953285 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:56.954220 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:56.954518 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:56.954522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:56.954787 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:56.955010 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:56.955014 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:56.955747 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:56.955747 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:56.956674 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:56.956676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:56.957091 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:56.957092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:56.957198 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:56.957788 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:56.957797 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:56.959304 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:56.961230 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:56.961240 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:56.961242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:56.961494 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:56.961721 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:56.961723 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:56.961723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:56.961847 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:56.962132 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:56.962162 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:56.962314 (+ 152us) client_negotiation.cc:769] Sending connection context
0812 14:22:56.962432 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 14:22:56.962608 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":159,"threads_started":1}
I20260812 14:22:56.963395 12648 heartbeater.cc:347] Connected to a master server at 127.5.148.62:34487
I20260812 14:22:56.963503 12648 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:56.963724 12648 heartbeater.cc:511] Master 127.5.148.62:34487 requested a full tablet report, sending...
I20260812 14:22:56.964182 12266 ts_manager.cc:194] Registered new tserver with Master: 24fe3a01a31840d2ad33a282ee62d99f (127.5.148.2:39865)
I20260812 14:22:56.964725 12266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:60089
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:22:56.989297 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:34487
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:41359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:22:57.053517 12492 heartbeater.cc:503] Master 127.5.148.62:34487 was elected leader, sending a full tablet report...
W20260812 14:22:57.081226 12655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:57.081444 12655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:57.081471 12655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:57.082666 12655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:57.082715 12655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:57.082768 12655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:22:57.084317 12655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:57.084585 12655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:57.084879 12655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:57.090309 12666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:57.090504 12661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:57.090327 12662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:57.090629 12655 server_base.cc:1034] running on GCE node
I20260812 14:22:57.090823 12655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:57.091102 12655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:57.092243 12655 hybrid_clock.cc:648] HybridClock initialized: now 1786544577092222 us; error 37 us; skew 500 ppm
Aug 12 14:22:57 dist-test-slave-8w9v krb5kdc[12207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544577, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:57.095178 12655 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:22:57.095657 12655 webserver.cc:460] Webserver started at http://127.5.148.3:40611/ using document root <none> and password file <none>
I20260812 14:22:57.095878 12655 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:57.095921 12655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:57.096028 12655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:57.097150 12655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "0a726d20fef64f7f81ab434bbb524ff3"
format_stamp: "Formatted at 2026-08-12 14:22:57 on dist-test-slave-8w9v"
server_key: "fcdeeb35617215331173579c63c7f39d"
server_key_iv: "82375a4ff0da7f743d18f925bdab029e"
server_key_version: "encryptionkey@0"
I20260812 14:22:57.097469 12655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "0a726d20fef64f7f81ab434bbb524ff3"
format_stamp: "Formatted at 2026-08-12 14:22:57 on dist-test-slave-8w9v"
server_key: "fcdeeb35617215331173579c63c7f39d"
server_key_iv: "82375a4ff0da7f743d18f925bdab029e"
server_key_version: "encryptionkey@0"
I20260812 14:22:57.098827 12655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:22:57.099623 12675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:57.099844 12655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:57.099942 12655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "0a726d20fef64f7f81ab434bbb524ff3"
format_stamp: "Formatted at 2026-08-12 14:22:57 on dist-test-slave-8w9v"
server_key: "fcdeeb35617215331173579c63c7f39d"
server_key_iv: "82375a4ff0da7f743d18f925bdab029e"
server_key_version: "encryptionkey@0"
I20260812 14:22:57.100008 12655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:57.127032 12655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:57.129587 12655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:57.129714 12655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:57.129927 12655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:57.130281 12655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:57.130321 12655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:57.130343 12655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:57.130357 12655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:57.138007 12655 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:38955
I20260812 14:22:57.138104 12801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:38955 every 8 connection(s)
I20260812 14:22:57.138440 12655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:22:57 dist-test-slave-8w9v krb5kdc[12207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544577, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:22:57.145897 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 12655
I20260812 14:22:57.146030 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:22:57.146214 5712 external_mini_cluster.cc:1428] Setting key d6f4c11f4b583f193b597db649edd9b7
I20260812 14:22:57.149013 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:57.139330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:48315 (local address 127.5.148.62:34487)
0812 14:22:57.139403 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:57.139405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:57.139646 (+ 241us) server_negotiation.cc:407] Connection header received
0812 14:22:57.140779 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:57.140781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:57.140799 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:57.140857 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:57.141614 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:57.142146 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:57.143297 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:57.143518 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:57.143598 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:57.145955 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:57.145956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:57.145957 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:57.145975 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:57.147707 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:57.148103 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:57.148104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:57.148104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:57.148222 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:57.148470 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:57.148470 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:57.148471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:57.148581 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:57.148635 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:57.148872 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 14:22:57.148946 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 14:22:57.149214 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:57.139294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:34487 (local address 127.5.148.3:48315)
0812 14:22:57.139514 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:57.139553 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:57.140654 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:57.140934 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:57.140939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:57.141221 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:57.141506 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:57.141511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:57.142257 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:57.142258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:57.143190 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:57.143193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:57.143619 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:57.143620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:57.143792 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:57.144399 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:57.144408 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:57.145851 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:57.147789 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:57.147796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:57.147798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:57.148036 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:57.148292 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:57.148293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:57.148293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:57.148403 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:57.148640 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:57.148644 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:57.148763 (+ 119us) client_negotiation.cc:769] Sending connection context
0812 14:22:57.148862 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 14:22:57.148991 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":97,"threads_started":1}
I20260812 14:22:57.149658 12802 heartbeater.cc:347] Connected to a master server at 127.5.148.62:34487
I20260812 14:22:57.149744 12802 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:57.149897 12802 heartbeater.cc:511] Master 127.5.148.62:34487 requested a full tablet report, sending...
I20260812 14:22:57.150238 12266 ts_manager.cc:194] Registered new tserver with Master: 0a726d20fef64f7f81ab434bbb524ff3 (127.5.148.3:38955)
I20260812 14:22:57.150631 12266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:48315
I20260812 14:22:57.158552 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/test-admin.keytab.
Aug 12 14:22:57 dist-test-slave-8w9v krb5kdc[12207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:57.177045 5712 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 14:22:57.178756 12809 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:22:57.177968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35478 (local address 127.5.148.1:33069)
0812 14:22:57.178137 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:57.178142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:57.178152 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:22:57.178181 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:57.178183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:57.178197 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:57.178271 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:57.178666 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35478: BlockingRecv error: recv got EOF from 127.0.0.1:35478 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
W20260812 14:22:57.178870 5712 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/krb5cc' of type FILE
Aug 12 14:22:57 dist-test-slave-8w9v krb5kdc[12207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:57.179946 5712 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 14:22:57 dist-test-slave-8w9v krb5kdc[12207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:57.180965 5712 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 14:22:57 dist-test-slave-8w9v krb5kdc[12207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.1@KRBTEST.COM
I20260812 14:22:57.189208 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:57.181540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.5.148.1:33069)
0812 14:22:57.181602 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:57.181604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:57.181618 (+ 14us) server_negotiation.cc:407] Connection header received
0812 14:22:57.181719 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:57.181720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:57.181739 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:57.181803 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:57.182231 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:57.182951 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:57.183679 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:57.184017 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:57.184111 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:57.186262 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:57.186273 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:57.186274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:57.186283 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:57.187862 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:57.188352 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:57.188356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:57.188356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:57.188420 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:57.188630 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:57.188631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:57.188632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:57.188788 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:57.188962 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:57.189013 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 14:22:57.189101 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260812 14:22:57.191294 12809 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:22:57.190577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.5.148.1:33069)
0812 14:22:57.190633 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:57.190635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:57.190684 (+ 49us) server_negotiation.cc:407] Connection header received
0812 14:22:57.190758 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:57.190759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:57.190776 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:57.190839 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:57.191259 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35506: BlockingRecv error: recv got EOF from 127.0.0.1:35506 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260812 14:22:57.191533 5712 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/krb5cc' of type FILE
Aug 12 14:22:57 dist-test-slave-8w9v krb5kdc[12207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:57.192801 5712 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 14:22:57 dist-test-slave-8w9v krb5kdc[12207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:57.193743 5712 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 14:22:57 dist-test-slave-8w9v krb5kdc[12207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.1@KRBTEST.COM
I20260812 14:22:57.201247 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:57.194356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.5.148.1:33069)
0812 14:22:57.194435 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:57.194437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:57.194496 (+ 59us) server_negotiation.cc:407] Connection header received
0812 14:22:57.194558 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:57.194558 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:57.194571 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:57.194638 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:57.195018 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:57.195507 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:57.196256 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:57.196466 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:57.196542 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:57.198605 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:57.198607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:57.198607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:57.198613 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:57.200050 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:57.200480 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:57.200481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:57.200481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:57.200613 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:57.200831 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:57.200832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:57.200833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:57.200944 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:57.201036 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:57.201120 (+ 84us) server_negotiation.cc:299] Negotiation successful
0812 14:22:57.201184 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260812 14:22:57.202916 12809 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:22:57.202326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.5.148.1:33069)
0812 14:22:57.202428 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:57.202430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:57.202463 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:22:57.202514 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:57.202515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:57.202522 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:57.202567 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:57.202868 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35532: BlockingRecv error: recv got EOF from 127.0.0.1:35532 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 14:22:57.203074 5712 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786544547303129-5712-0/krb5kdc/krb5cc)
Aug 12 14:22:57 dist-test-slave-8w9v krb5kdc[12207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:57.204272 5712 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 14:22:57 dist-test-slave-8w9v krb5kdc[12207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:57.205253 5712 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 14:22:57 dist-test-slave-8w9v krb5kdc[12207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.1@KRBTEST.COM
I20260812 14:22:57.212368 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:57.205812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.5.148.1:33069)
0812 14:22:57.205848 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:57.205850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:57.205962 (+ 112us) server_negotiation.cc:407] Connection header received
0812 14:22:57.206040 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:57.206041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:57.206050 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:57.206116 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:57.206506 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:57.206971 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:57.207660 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:57.207851 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:57.207943 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:57.210012 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:57.210013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:57.210014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:57.210020 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:57.211177 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:57.211484 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:57.211485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:57.211485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:57.211534 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:57.211816 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:57.211816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:57.211817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:57.211903 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:57.211973 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:57.212081 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 14:22:57.212248 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:22:57.213001 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 12338
I20260812 14:22:57.226861 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 12500
I20260812 14:22:57.233630 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 12655
I20260812 14:22:57.241671 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 12225
2026-08-12T14:22:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3563 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:22:57 dist-test-slave-8w9v krb5kdc[12829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:22:57 dist-test-slave-8w9v krb5kdc[12829](info): set up 2 sockets
Aug 12 14:22:57 dist-test-slave-8w9v krb5kdc[12829](info): commencing operation
krb5kdc: starting...
W20260812 14:22:58.378110 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:22:58 dist-test-slave-8w9v krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:22:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:22:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.62.keytab.
Entry for principal oryx/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.62.keytab.
I20260812 14:22:58.519433 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:33907
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.62.keytab
--principal=oryx/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:33907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:58.606276 12843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:58.606567 12843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:58.606611 12843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:58.607817 12843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:58.607903 12843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:58.607928 12843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:58.607949 12843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:58.607981 12843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:22:58.609539 12843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:33907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.62.keytab
--principal=oryx/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:33907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:58.609846 12843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:58.610098 12843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:58.615357 12852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:58.615343 12849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:58.615317 12850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:58.615944 12843 server_base.cc:1034] running on GCE node
I20260812 14:22:58.616305 12843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:58.616601 12843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:58.617784 12843 hybrid_clock.cc:648] HybridClock initialized: now 1786544578617763 us; error 39 us; skew 500 ppm
Aug 12 14:22:58 dist-test-slave-8w9v krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544578, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:58.620658 12843 init.cc:377] Logged in from keytab as oryx/127.5.148.62@KRBTEST.COM (short username oryx)
I20260812 14:22:58.621217 12843 webserver.cc:460] Webserver started at http://127.5.148.62:40677/ using document root <none> and password file <none>
I20260812 14:22:58.621474 12843 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:58.621546 12843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:58.621724 12843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:58.622910 12843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "8bac19a5015e4f9085a7a1c559f4a309"
format_stamp: "Formatted at 2026-08-12 14:22:58 on dist-test-slave-8w9v"
server_key: "631f34238ca9553eba52ea37fdf8fef1"
server_key_iv: "a10a494237b7ac2cf65e56f163ffd4c0"
server_key_version: "encryptionkey@0"
I20260812 14:22:58.623317 12843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "8bac19a5015e4f9085a7a1c559f4a309"
format_stamp: "Formatted at 2026-08-12 14:22:58 on dist-test-slave-8w9v"
server_key: "631f34238ca9553eba52ea37fdf8fef1"
server_key_iv: "a10a494237b7ac2cf65e56f163ffd4c0"
server_key_version: "encryptionkey@0"
I20260812 14:22:58.624912 12843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:22:58.625759 12861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:58.625943 12843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:58.626065 12843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "8bac19a5015e4f9085a7a1c559f4a309"
format_stamp: "Formatted at 2026-08-12 14:22:58 on dist-test-slave-8w9v"
server_key: "631f34238ca9553eba52ea37fdf8fef1"
server_key_iv: "a10a494237b7ac2cf65e56f163ffd4c0"
server_key_version: "encryptionkey@0"
I20260812 14:22:58.626150 12843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:58.646787 12843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:58.648901 12843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:58.649086 12843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:58.653362 12843 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:33907
I20260812 14:22:58.653448 12925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:33907 every 8 connection(s)
I20260812 14:22:58.653810 12843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:22:58.654855 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:58.655897 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 12843
I20260812 14:22:58.656038 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:22:58.656266 5712 external_mini_cluster.cc:1428] Setting key 49351e09a6837f149078c01dd7d2d4db
I20260812 14:22:58.657845 12926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309: Bootstrap starting.
I20260812 14:22:58.658490 12926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:58.658792 12926 log.cc:824] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:58.659569 12926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309: No bootstrap required, opened a new log
I20260812 14:22:58.661489 12926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } }
I20260812 14:22:58.661656 12926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:58.661717 12926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bac19a5015e4f9085a7a1c559f4a309, State: Initialized, Role: FOLLOWER
I20260812 14:22:58.661895 12926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } }
I20260812 14:22:58.662005 12926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:22:58.662053 12926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:22:58.662103 12926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:58.662689 12926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } }
I20260812 14:22:58.662811 12926 leader_election.cc:302] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bac19a5015e4f9085a7a1c559f4a309; no voters:
I20260812 14:22:58.663002 12926 leader_election.cc:288] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:22:58.663030 12932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:22:58.663146 12932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [term 1 LEADER]: Becoming Leader. State: Replica: 8bac19a5015e4f9085a7a1c559f4a309, State: Running, Role: LEADER
I20260812 14:22:58.663245 12932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } }
I20260812 14:22:58.663292 12926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 14:22:58 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
I20260812 14:22:58.663616 12932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bac19a5015e4f9085a7a1c559f4a309. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } }
I20260812 14:22:58.663686 12932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:58.664381 12933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } }
I20260812 14:22:58.664474 12933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:58.664918 12941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:22:58.667943 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:58.656838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.5.148.62:33907)
0812 14:22:58.657057 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:58.657067 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:58.657101 (+ 34us) server_negotiation.cc:407] Connection header received
0812 14:22:58.657659 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:58.657666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:58.657714 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:58.657889 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:58.658544 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:58.659767 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:58.660638 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:58.661058 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:58.661177 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:58.664083 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:58.664088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:58.664089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:58.664104 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:58.666259 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:58.666683 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:58.666685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:58.666686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:58.666800 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:58.667046 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:58.667047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:58.667047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:58.667310 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:58.667426 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:58.667580 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 14:22:58.667686 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":102,"threads_started":1}
I20260812 14:22:58.668280 12941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:22:58.670482 12941 catalog_manager.cc:1347] Generated new cluster ID: a52194445a7e4e6ca42c123286c3fa53
I20260812 14:22:58.670528 12941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:22:58.679131 12941 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:22:58.679854 12941 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:22:58.683857 12941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309: Generated new TSK 0
I20260812 14:22:58.684196 12941 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.1.keytab.
Entry for principal oryx/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.1.keytab.
I20260812 14:22:58.720144 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:33907
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.1.keytab
--principal=oryx/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:58.809978 12962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:58.810266 12962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:58.810326 12962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:58.811558 12962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:58.811632 12962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:58.811681 12962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:22:58.813354 12962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.1.keytab
--principal=oryx/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:33907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:58.813664 12962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:58.813927 12962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:58.819564 12972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:58.819552 12968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:58.819545 12969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:58.819954 12962 server_base.cc:1034] running on GCE node
I20260812 14:22:58.820217 12962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:58.820483 12962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:58.821668 12962 hybrid_clock.cc:648] HybridClock initialized: now 1786544578821654 us; error 32 us; skew 500 ppm
Aug 12 14:22:58 dist-test-slave-8w9v krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544578, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:58.824590 12962 init.cc:377] Logged in from keytab as oryx/127.5.148.1@KRBTEST.COM (short username oryx)
I20260812 14:22:58.825094 12962 webserver.cc:460] Webserver started at http://127.5.148.1:40715/ using document root <none> and password file <none>
I20260812 14:22:58.825361 12962 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:58.825408 12962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:58.825558 12962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:58.826745 12962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054"
format_stamp: "Formatted at 2026-08-12 14:22:58 on dist-test-slave-8w9v"
server_key: "1e4a25fa8a34686aef970433de994164"
server_key_iv: "3b9a1a1b7035b4a1d0f51e40509a92b5"
server_key_version: "encryptionkey@0"
I20260812 14:22:58.827133 12962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054"
format_stamp: "Formatted at 2026-08-12 14:22:58 on dist-test-slave-8w9v"
server_key: "1e4a25fa8a34686aef970433de994164"
server_key_iv: "3b9a1a1b7035b4a1d0f51e40509a92b5"
server_key_version: "encryptionkey@0"
I20260812 14:22:58.828750 12962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:22:58.829769 12979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:58.829977 12962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:58.830080 12962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054"
format_stamp: "Formatted at 2026-08-12 14:22:58 on dist-test-slave-8w9v"
server_key: "1e4a25fa8a34686aef970433de994164"
server_key_iv: "3b9a1a1b7035b4a1d0f51e40509a92b5"
server_key_version: "encryptionkey@0"
I20260812 14:22:58.830161 12962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:58.843339 12962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:58.846258 12962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:58.846426 12962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:58.846706 12962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:58.847069 12962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:58.847137 12962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:58.847184 12962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:58.847224 12962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:58.854212 12962 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:43061
I20260812 14:22:58.854300 13120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:43061 every 8 connection(s)
I20260812 14:22:58.854656 12962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:22:58.857185 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 12962
I20260812 14:22:58.857296 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:22:58.857506 5712 external_mini_cluster.cc:1428] Setting key 34600fd0a01e4240c5bd2e19f4b36b4e
Aug 12 14:22:58 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544578, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
I20260812 14:22:58.865235 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:58.855442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:43709 (local address 127.5.148.62:33907)
0812 14:22:58.855527 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:58.855530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:58.855719 (+ 189us) server_negotiation.cc:407] Connection header received
0812 14:22:58.856374 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:58.856375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:58.856387 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:58.856457 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:58.857821 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:58.858424 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:58.859448 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:58.859680 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:58.859761 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:58.862036 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:58.862037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:58.862038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:58.862046 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:58.863641 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:58.864157 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:58.864158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:58.864159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:58.864221 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:58.864526 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:58.864527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:58.864528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:58.864664 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:58.864746 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:58.865047 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 14:22:58.865140 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 14:22:58.865442 13123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:58.855394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.1:43709)
0812 14:22:58.855592 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:58.855631 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:58.856278 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:58.856520 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:58.856523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:58.857330 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:58.857701 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:58.857711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:58.858531 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:58.858532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:58.859344 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:58.859346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:58.859770 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:58.859772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:58.859902 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:58.860518 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:58.860526 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:58.861924 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:58.863763 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:58.863773 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:58.863775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:58.864027 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:58.864300 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:58.864301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:58.864302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:58.864466 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:58.864735 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:58.864743 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:58.864903 (+ 160us) client_negotiation.cc:769] Sending connection context
0812 14:22:58.865011 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 14:22:58.865182 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":126,"threads_started":1}
I20260812 14:22:58.866407 13121 heartbeater.cc:347] Connected to a master server at 127.5.148.62:33907
I20260812 14:22:58.866521 13121 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:58.866734 13121 heartbeater.cc:511] Master 127.5.148.62:33907 requested a full tablet report, sending...
I20260812 14:22:58.867344 12884 ts_manager.cc:194] Registered new tserver with Master: 8bbd51a2cc8c4fe6af24d4ffda7f4054 (127.5.148.1:43061)
WARNING: no policy specified for oryx/127.5.148.2@KRBTEST.COM; defaulting to no policy
I20260812 14:22:58.868049 12884 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:43709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.2.keytab.
Entry for principal oryx/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.2.keytab.
I20260812 14:22:58.889025 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:33907
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.2.keytab
--principal=oryx/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:58.985512 13126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:58.985735 13126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:58.985762 13126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:58.986994 13126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:58.987042 13126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:58.987071 13126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:22:58.988821 13126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.2.keytab
--principal=oryx/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:33907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:58.989051 13126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:58.989250 13126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:58.994827 13135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:58.994836 13139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:58.994901 13126 server_base.cc:1034] running on GCE node
W20260812 14:22:58.994846 13134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:58.995350 13126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:58.995659 13126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:58.996824 13126 hybrid_clock.cc:648] HybridClock initialized: now 1786544578996812 us; error 45 us; skew 500 ppm
Aug 12 14:22:58 dist-test-slave-8w9v krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544578, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:58.999724 13126 init.cc:377] Logged in from keytab as oryx/127.5.148.2@KRBTEST.COM (short username oryx)
I20260812 14:22:59.000340 13126 webserver.cc:460] Webserver started at http://127.5.148.2:46463/ using document root <none> and password file <none>
I20260812 14:22:59.000598 13126 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:59.000669 13126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:59.000825 13126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:59.001991 13126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "dec94ea6b5974186b3f220955b548336"
format_stamp: "Formatted at 2026-08-12 14:22:58 on dist-test-slave-8w9v"
server_key: "a7db0a781a7e405463c7e88fb594950c"
server_key_iv: "b0203f2906003a6145366be139b5eb9d"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.002389 13126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "dec94ea6b5974186b3f220955b548336"
format_stamp: "Formatted at 2026-08-12 14:22:58 on dist-test-slave-8w9v"
server_key: "a7db0a781a7e405463c7e88fb594950c"
server_key_iv: "b0203f2906003a6145366be139b5eb9d"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.003890 13126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:22:59.004920 13147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.005158 13126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:59.005277 13126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "dec94ea6b5974186b3f220955b548336"
format_stamp: "Formatted at 2026-08-12 14:22:58 on dist-test-slave-8w9v"
server_key: "a7db0a781a7e405463c7e88fb594950c"
server_key_iv: "b0203f2906003a6145366be139b5eb9d"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.005349 13126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:59.021055 13126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:59.023325 13126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:59.023512 13126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:59.023774 13126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:59.024111 13126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:59.024228 13126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.024273 13126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:59.024302 13126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.032980 13126 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:38119
I20260812 14:22:59.033033 13283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:38119 every 8 connection(s)
I20260812 14:22:59.033465 13126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:22:59.036092 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 13126
I20260812 14:22:59.036250 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:22:59.036466 5712 external_mini_cluster.cc:1428] Setting key 8df1205230546a7e49edc2a59fbebf26
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544578, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
I20260812 14:22:59.045799 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.034368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:58669 (local address 127.5.148.62:33907)
0812 14:22:59.034438 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.034440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.034881 (+ 441us) server_negotiation.cc:407] Connection header received
0812 14:22:59.035513 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.035514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.035528 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.035590 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.036916 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.037525 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.038607 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.038937 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.039015 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.042202 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.042204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.042204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.042214 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.044214 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.044656 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.044658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.044658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.044732 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.045032 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.045034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.045034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.045204 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.045285 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.045573 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.045686 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 14:22:59.045890 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.034366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.2:58669)
0812 14:22:59.034567 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.034607 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.035366 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.035669 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.035673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.036463 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.036802 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.036810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.037652 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.037653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.038503 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.038506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.039037 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.039037 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.039320 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.039996 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.040010 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.042088 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.044311 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.044318 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.044320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.044570 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.044810 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.044811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.044811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.044957 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.045284 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.045289 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.045425 (+ 136us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.045517 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.045655 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
I20260812 14:22:59.046505 13284 heartbeater.cc:347] Connected to a master server at 127.5.148.62:33907
I20260812 14:22:59.046605 13284 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:59.046813 13284 heartbeater.cc:511] Master 127.5.148.62:33907 requested a full tablet report, sending...
I20260812 14:22:59.047299 12884 ts_manager.cc:194] Registered new tserver with Master: dec94ea6b5974186b3f220955b548336 (127.5.148.2:38119)
I20260812 14:22:59.047863 12884 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:58669
WARNING: no policy specified for oryx/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.3.keytab.
Entry for principal oryx/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.3.keytab.
I20260812 14:22:59.068277 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:33907
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.3.keytab
--principal=oryx/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:59.167105 13292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:59.167349 13292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:59.167376 13292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:59.168686 13292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:59.168767 13292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:59.168798 13292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:22:59.170364 13292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.3.keytab
--principal=oryx/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:33907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:59.170619 13292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:59.170868 13292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:59.176474 13298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:59.176474 13297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:59.176507 13301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:59.176661 13292 server_base.cc:1034] running on GCE node
I20260812 14:22:59.177027 13292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:59.177327 13292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:59.178511 13292 hybrid_clock.cc:648] HybridClock initialized: now 1786544579178514 us; error 51 us; skew 500 ppm
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544579, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:59.181440 13292 init.cc:377] Logged in from keytab as oryx/127.5.148.3@KRBTEST.COM (short username oryx)
I20260812 14:22:59.181917 13292 webserver.cc:460] Webserver started at http://127.5.148.3:37905/ using document root <none> and password file <none>
I20260812 14:22:59.182157 13292 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:59.182229 13292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:59.182387 13292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:59.183513 13292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "62540953877147abab1c3c4206ff0bc7"
format_stamp: "Formatted at 2026-08-12 14:22:59 on dist-test-slave-8w9v"
server_key: "b9601f574aa4fb0791b2aa710b987231"
server_key_iv: "06ad6210c262e66eede751924aec8e36"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.183938 13292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "62540953877147abab1c3c4206ff0bc7"
format_stamp: "Formatted at 2026-08-12 14:22:59 on dist-test-slave-8w9v"
server_key: "b9601f574aa4fb0791b2aa710b987231"
server_key_iv: "06ad6210c262e66eede751924aec8e36"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.185449 13292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:59.186254 13309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.186451 13292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:59.186575 13292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "62540953877147abab1c3c4206ff0bc7"
format_stamp: "Formatted at 2026-08-12 14:22:59 on dist-test-slave-8w9v"
server_key: "b9601f574aa4fb0791b2aa710b987231"
server_key_iv: "06ad6210c262e66eede751924aec8e36"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.186654 13292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:59.207147 13292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:59.209937 13292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:59.210114 13292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:59.210384 13292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:59.210724 13292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:59.210783 13292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.210830 13292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:59.210868 13292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.217370 13292 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:39507
I20260812 14:22:59.217502 13443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:39507 every 8 connection(s)
I20260812 14:22:59.217808 13292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544579, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
I20260812 14:22:59.224973 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 13292
I20260812 14:22:59.225086 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:22:59.225313 5712 external_mini_cluster.cc:1428] Setting key 934a357d608ed12dbb98805b21b2581b
I20260812 14:22:59.227648 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.218607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:49825 (local address 127.5.148.62:33907)
0812 14:22:59.218684 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.218686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.218914 (+ 228us) server_negotiation.cc:407] Connection header received
0812 14:22:59.219423 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.219425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.219436 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.219496 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.220579 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.221086 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.222161 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.222348 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.222404 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.224662 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.224664 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.224664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.224671 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.226497 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.226889 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.226890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.226890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.226943 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.227125 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.227126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.227126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.227228 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.227277 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.227518 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.227582 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 14:22:59.227854 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.218581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.3:49825)
0812 14:22:59.218768 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.218817 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.219328 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.219561 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.219564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.220267 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.220486 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.220492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.221179 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.221179 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.222082 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.222085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.222446 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.222447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.222552 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.223052 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.223061 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.224567 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.226566 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.226573 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.226575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.226813 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.226985 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.226986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.226986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.227076 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.227268 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.227270 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.227389 (+ 119us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.227488 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.227619 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260812 14:22:59.228446 13444 heartbeater.cc:347] Connected to a master server at 127.5.148.62:33907
I20260812 14:22:59.228575 13444 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:59.228767 13444 heartbeater.cc:511] Master 127.5.148.62:33907 requested a full tablet report, sending...
I20260812 14:22:59.229206 12884 ts_manager.cc:194] Registered new tserver with Master: 62540953877147abab1c3c4206ff0bc7 (127.5.148.3:39507)
I20260812 14:22:59.229660 12884 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:49825
I20260812 14:22:59.237682 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.148.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.148.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.148.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.61.keytab.
Entry for principal oryx/127.5.148.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.61.keytab.
I20260812 14:22:59.268095 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.61:35517
--webserver_interface=127.5.148.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.61.keytab
--principal=oryx/127.5.148.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:33907,127.5.148.61:35517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:59.354094 13450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:59.354323 13450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:59.354347 13450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:59.355707 13450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:59.355758 13450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:59.355773 13450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:59.355786 13450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:59.355799 13450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:22:59.357393 13450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:33907,127.5.148.61:35517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.61.keytab
--principal=oryx/127.5.148.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.61:35517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.148.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
W20260812 14:22:59.357534 13450 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.148.62:33907,127.5.148.61:35517'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 14:22:59.357698 13450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:59.357959 13450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:59.362991 13458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:59.363032 13456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:59.363020 13455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:59.363402 13450 server_base.cc:1034] running on GCE node
I20260812 14:22:59.363727 13450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:59.364073 13450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:59.365259 13450 hybrid_clock.cc:648] HybridClock initialized: now 1786544579365238 us; error 39 us; skew 500 ppm
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544579, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:59.368098 13450 init.cc:377] Logged in from keytab as oryx/127.5.148.61@KRBTEST.COM (short username oryx)
I20260812 14:22:59.368583 13450 webserver.cc:460] Webserver started at http://127.5.148.61:34709/ using document root <none> and password file <none>
I20260812 14:22:59.368846 13450 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:59.368896 13450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:59.369036 13450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:59.370150 13450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/data/instance:
uuid: "36dc93bfb0624fe8af2565d5bd070107"
format_stamp: "Formatted at 2026-08-12 14:22:59 on dist-test-slave-8w9v"
server_key: "35f0c63f5d909b9447c558b37c048cd3"
server_key_iv: "588d0c61672dc11100e61557d887ef97"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.370532 13450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/wal/instance:
uuid: "36dc93bfb0624fe8af2565d5bd070107"
format_stamp: "Formatted at 2026-08-12 14:22:59 on dist-test-slave-8w9v"
server_key: "35f0c63f5d909b9447c558b37c048cd3"
server_key_iv: "588d0c61672dc11100e61557d887ef97"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.371944 13450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:22:59.372895 13465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.373096 13450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:59.373211 13450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/wal
uuid: "36dc93bfb0624fe8af2565d5bd070107"
format_stamp: "Formatted at 2026-08-12 14:22:59 on dist-test-slave-8w9v"
server_key: "35f0c63f5d909b9447c558b37c048cd3"
server_key_iv: "588d0c61672dc11100e61557d887ef97"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.373293 13450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:59.387445 13450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:59.390062 13450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:59.390244 13450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:59.394613 13450 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.61:35517
I20260812 14:22:59.394670 13526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.61:35517 every 8 connection(s)
I20260812 14:22:59.395085 13450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/data/info.pb
I20260812 14:22:59.396237 13527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:59.398059 13527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544579, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.61@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
I20260812 14:22:59.404628 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 13450
I20260812 14:22:59.408553 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.398892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.5.148.62:33907)
0812 14:22:59.398967 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.398969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.399190 (+ 221us) server_negotiation.cc:407] Connection header received
0812 14:22:59.399745 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.399747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.399759 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.399806 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.400638 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.401193 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.402297 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.402487 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.402539 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.405035 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.405037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.405037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.405047 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.406827 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.407272 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.407274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.407274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.407351 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.407654 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.407655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.407656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.407871 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.407957 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.408359 (+ 402us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.408455 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 14:22:59.408780 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.398854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.0.0.1:55076)
0812 14:22:59.399056 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.399096 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.399589 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.399875 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.399879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.400241 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.400546 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.400553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.401342 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.401344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.402213 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.402216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.402568 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.402569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.402691 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.403214 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.403223 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.404880 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.406917 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.406927 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.406929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.407187 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.407448 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.407449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.407450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.407574 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.407986 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.407994 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.408184 (+ 190us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.408311 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.408514 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":5,"thread_start_us":119,"threads_started":1}
I20260812 14:22:59.409626 13527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.148.61@KRBTEST.COM
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544579, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.61@KRBTEST.COM for oryx/127.5.148.61@KRBTEST.COM
I20260812 14:22:59.414888 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.405340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.5.148.61:35517)
0812 14:22:59.405541 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.405546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.405567 (+ 21us) server_negotiation.cc:407] Connection header received
0812 14:22:59.405592 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.405595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.405609 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.405695 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.406521 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.407241 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.408308 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.408664 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.408742 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.411494 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.411503 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.411503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.411510 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.413515 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.413937 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.413941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.413941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.414007 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.414207 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.414209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.414209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.414430 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.414638 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.414697 (+ 59us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.414785 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":97,"threads_started":1}
I20260812 14:22:59.418246 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.410012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53776 (local address 127.5.148.61:35517)
0812 14:22:59.410194 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.410196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.410201 (+ 5us) server_negotiation.cc:407] Connection header received
0812 14:22:59.410215 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.410215 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.410223 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.410294 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.410618 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.411161 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.411926 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.412211 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.412282 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.414187 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.414188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.414189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.414194 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.416887 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.417306 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.417307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.417308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.417379 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.417604 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.417605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.417605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.417727 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.417816 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.417967 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.418085 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":150,"thread_start_us":123,"threads_started":1}
I20260812 14:22:59.418258 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.409975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35517 (local address 127.0.0.1:53776)
0812 14:22:59.410008 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.410018 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.410060 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.410311 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.410311 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.410470 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.410531 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.410533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.411247 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.411248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.411834 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.411836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.412294 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.412296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.412404 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.412742 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.412748 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.414107 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.417003 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.417004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.417004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.417189 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.417450 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.417452 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.417452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.417520 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.417805 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.417805 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.417846 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.417943 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.418152 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16,"mutex_wait_us":107}
I20260812 14:22:59.419780 13527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: Bootstrap starting.
I20260812 14:22:59.420572 13527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:59.420900 13527 log.cc:824] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:59.421718 13527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: No bootstrap required, opened a new log
I20260812 14:22:59.423403 13527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } }
I20260812 14:22:59.423559 13527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:59.423624 13527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36dc93bfb0624fe8af2565d5bd070107, State: Initialized, Role: FOLLOWER
I20260812 14:22:59.423769 13527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } }
I20260812 14:22:59.424080 13535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } } }
I20260812 14:22:59.424233 13535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:22:59.424248 13527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 14:22:59.426075 13547 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:22:59.426132 13547 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:22:59.428079 13450 master_runner.cc:428] Detected that this master 36dc93bfb0624fe8af2565d5bd070107 is joining an existing cluster
I20260812 14:22:59.428221 13450 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.148.61:35517
I20260812 14:22:59.437510 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.429516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35517 (local address 127.0.0.1:53786)
0812 14:22:59.429668 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.429678 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.429771 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.429992 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.429992 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.430276 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.430346 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.430349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.431324 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.431324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.431894 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.431896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.432494 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.432495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.432555 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.433122 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.433127 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.433676 (+ 549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.435966 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.435967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.435968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.436391 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.436630 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.436631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.436631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.436684 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.437030 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.437031 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.437090 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.437215 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.437357 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260812 14:22:59.437553 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.429424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.5.148.62:33907)
0812 14:22:59.429525 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.429527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.429632 (+ 105us) server_negotiation.cc:407] Connection header received
0812 14:22:59.429751 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.429752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.429767 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.429838 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.430424 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.431223 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.432388 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.432685 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.432775 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.434181 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.434183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.434183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.434191 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.436180 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.436614 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.436616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.436616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.436689 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.436946 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.436948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.436948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.437095 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.437172 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.437360 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.437454 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 14:22:59.437522 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.429357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.0.0.1:55090)
0812 14:22:59.429561 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.429573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.429664 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.429852 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.429853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.430208 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.430287 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.430289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.431326 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.431327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.432292 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.432295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.433119 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.433120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.433192 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.433658 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.433664 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.434055 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.436254 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.436255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.436256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.436520 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.436786 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.436787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.436787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.436866 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.437174 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.437175 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.437215 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.437324 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.437409 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
I20260812 14:22:59.437784 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.429408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.5.148.61:35517)
0812 14:22:59.429482 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.429484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.429747 (+ 263us) server_negotiation.cc:407] Connection header received
0812 14:22:59.429868 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.429869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.429878 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.429909 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.430462 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.431222 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.432003 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.432313 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.432399 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.433801 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.433802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.433803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.433810 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.435856 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.436480 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.436482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.436482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.436549 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.436764 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.436765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.436765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.436940 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.437038 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.437251 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.437698 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":355,"server-negotiator.queue_time_us":47}
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544578, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.62@KRBTEST.COM for oryx/127.5.148.61@KRBTEST.COM
I20260812 14:22:59.448478 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.440552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53800 (local address 127.5.148.61:35517)
0812 14:22:59.440607 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.440609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.440810 (+ 201us) server_negotiation.cc:407] Connection header received
0812 14:22:59.440888 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.440889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.440897 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.440945 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.441452 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.441913 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.443046 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.443216 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.443266 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.445567 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.445573 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.445574 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.445580 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.447058 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.447525 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.447526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.447527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.447592 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.447840 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.447841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.447841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.447976 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.448050 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.448280 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.448370 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 14:22:59.448490 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.440472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35517 (local address 127.0.0.1:53800)
0812 14:22:59.440681 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.440738 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.440819 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.440952 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.440954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.441310 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.441374 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.441376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.441985 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.441986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.442961 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.442964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.443288 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.443289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.443377 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.443839 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.443844 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.445454 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.447175 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.447184 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.447187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.447438 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.447682 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.447683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.447684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.447751 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.448054 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.448063 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.448142 (+ 79us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.448229 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.448373 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":104,"threads_started":1}
I20260812 14:22:59.449258 12884 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.148.61:35517: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107"
member_type: NON_VOTER
last_known_addr {
host: "127.5.148.61"
port: 35517
}
attrs {
promote: true
}
}
dest_uuid: "8bac19a5015e4f9085a7a1c559f4a309"
cas_config_opid_index: -1
I20260812 14:22:59.449674 12884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } }
I20260812 14:22:59.450218 13558 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 36dc93bfb0624fe8af2565d5bd070107 (127.5.148.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } }
I20260812 14:22:59.450719 13497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:59.451442 13558 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.148.61:35517
I20260812 14:22:59.451768 13557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } }
I20260812 14:22:59.451849 13557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:59.451838 13450 master.cc:537] Master@127.5.148.61:35517 shutting down...
I20260812 14:22:59.452508 13497 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 FOLLOWER]: Refusing update from remote peer 8bac19a5015e4f9085a7a1c559f4a309: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 14:22:59.453200 13557 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 14:22:59.455045 12864 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260812 14:22:59.455075 13450 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:59.455165 13450 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:59.455219 13450 tablet_replica.cc:330] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: stopping tablet replica
W20260812 14:22:59.455716 12864 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 -> Peer 36dc93bfb0624fe8af2565d5bd070107 (127.5.148.61:35517): Couldn't send request to peer 36dc93bfb0624fe8af2565d5bd070107. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 14:22:59.468770 13450 master.cc:552] Master@127.5.148.61:35517 shutdown complete.
I20260812 14:22:59.468845 13450 master_runner.cc:305] Clearing existing system tablet
I20260812 14:22:59.468990 13450 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:59.469043 13450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:59.469888 13450 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:22:59.470551 13563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.470732 13450 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:59.470798 13450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:22:59.470824 13450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/wal
uuid: "36dc93bfb0624fe8af2565d5bd070107"
format_stamp: "Formatted at 2026-08-12 14:22:59 on dist-test-slave-8w9v"
server_key: "35f0c63f5d909b9447c558b37c048cd3"
server_key_iv: "588d0c61672dc11100e61557d887ef97"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.471907 13450 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:59.473820 13450 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 14:22:59.473915 13450 log.cc:1196] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 14:22:59.474131 13450 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: Deleting consensus metadata
I20260812 14:22:59.474283 13450 master_runner.cc:315] Copying system tablet from 127.5.148.62:33907
I20260812 14:22:59.474978 13450 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: tablet copy: Beginning tablet copy session from remote peer at address 127.5.148.62:33907
I20260812 14:22:59.482270 13569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.475452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.0.0.1:55106)
0812 14:22:59.475600 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.475612 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.475688 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.475868 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.475869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.476062 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.476179 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.476181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.477190 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.477191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.477782 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.477784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.478288 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.478289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.478349 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.478904 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.478909 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.479228 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.481235 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.481237 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.481237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.481440 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.481664 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.481665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.481665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.481720 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.481994 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.481995 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.482035 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.482088 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.482181 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":102,"threads_started":1}
I20260812 14:22:59.482321 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.475536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55106 (local address 127.5.148.62:33907)
0812 14:22:59.475595 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.475598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.475689 (+ 91us) server_negotiation.cc:407] Connection header received
0812 14:22:59.475766 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.475767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.475779 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.475842 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.476280 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.477090 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.477898 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.478208 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.478283 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.479327 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.479329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.479329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.479338 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.481139 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.481521 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.481523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.481523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.481592 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.481783 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.481783 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.481784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.481926 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.482000 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.482136 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.482221 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 14:22:59.482935 12906 tablet_copy_service.cc:140] P 8bac19a5015e4f9085a7a1c559f4a309: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 36dc93bfb0624fe8af2565d5bd070107 ({username='oryx', principal='oryx/127.5.148.61@KRBTEST.COM'} at 127.0.0.1:55106)
I20260812 14:22:59.483093 12906 tablet_copy_service.cc:161] P 8bac19a5015e4f9085a7a1c559f4a309: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 36dc93bfb0624fe8af2565d5bd070107 at {username='oryx', principal='oryx/127.5.148.61@KRBTEST.COM'} at 127.0.0.1:55106: session id = 36dc93bfb0624fe8af2565d5bd070107-00000000000000000000000000000000
I20260812 14:22:59.483876 12906 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 14:22:59.484669 13450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 14:22:59.486151 13450 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: tablet copy: Starting download of 0 data blocks...
I20260812 14:22:59.486323 13450 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: tablet copy: Starting download of 1 WAL segments...
I20260812 14:22:59.487449 13450 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 14:22:59.488646 12906 tablet_copy_service.cc:342] P 8bac19a5015e4f9085a7a1c559f4a309: Request end of tablet copy session 36dc93bfb0624fe8af2565d5bd070107-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.148.61@KRBTEST.COM'} at 127.0.0.1:55106
I20260812 14:22:59.488741 12906 tablet_copy_service.cc:433] P 8bac19a5015e4f9085a7a1c559f4a309: ending tablet copy session 36dc93bfb0624fe8af2565d5bd070107-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 36dc93bfb0624fe8af2565d5bd070107
I20260812 14:22:59.492079 13450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:59.493515 13571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:59.493536 13572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:59.493562 13575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:59.493745 13450 server_base.cc:1034] running on GCE node
I20260812 14:22:59.493880 13450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:59.494066 13450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:59.495213 13450 hybrid_clock.cc:648] HybridClock initialized: now 1786544579495196 us; error 36 us; skew 500 ppm
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544579, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:59.496938 13450 init.cc:377] Logged in from keytab as oryx/127.5.148.61@KRBTEST.COM (short username oryx)
I20260812 14:22:59.497308 13450 webserver.cc:460] Webserver started at http://127.5.148.61:34113/ using document root <none> and password file <none>
I20260812 14:22:59.497478 13450 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:59.497530 13450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:59.498338 13450 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:22:59.498944 13583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.499120 13450 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:22:59.499179 13450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/wal
uuid: "36dc93bfb0624fe8af2565d5bd070107"
format_stamp: "Formatted at 2026-08-12 14:22:59 on dist-test-slave-8w9v"
server_key: "35f0c63f5d909b9447c558b37c048cd3"
server_key_iv: "588d0c61672dc11100e61557d887ef97"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.499240 13450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:59.505473 13450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:59.507576 13450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:59.511279 13450 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.61:35517
I20260812 14:22:59.511718 13643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.61:35517 every 8 connection(s)
I20260812 14:22:59.511777 13450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-1/data/info.pb
I20260812 14:22:59.512647 13644 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:22:59.513108 13644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: Bootstrap starting.
I20260812 14:22:59.515722 13644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:22:59.515981 13644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: Bootstrap complete.
I20260812 14:22:59.516302 13644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } }
I20260812 14:22:59.516410 13644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 36dc93bfb0624fe8af2565d5bd070107, State: Initialized, Role: LEARNER
I20260812 14:22:59.516461 13644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } }
I20260812 14:22:59.516644 13644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:59.518154 13647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } }
I20260812 14:22:59.518230 13647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: This master's current role is: LEARNER
I20260812 14:22:59.518500 13661 catalog_manager.cc:1259] Loaded cluster ID: a52194445a7e4e6ca42c123286c3fa53
I20260812 14:22:59.518548 13661 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: loading cluster ID for follower catalog manager: success
I20260812 14:22:59.519500 13661 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: acquiring CA information for follower catalog manager: success
I20260812 14:22:59.519796 13661 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:22:59.525044 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.516766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.5.148.61:35517)
0812 14:22:59.516934 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.516937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.516970 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:22:59.517053 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.517054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.517066 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.517159 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.517941 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.518927 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.519917 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.520264 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.520344 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.521898 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.521899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.521900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.521908 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.523660 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.524078 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.524079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.524079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.524195 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.524395 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.524396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.524396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.524551 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.524669 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.524814 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.524922 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for oryx/127.5.148.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.148.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.148.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.60.keytab.
Entry for principal oryx/127.5.148.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.60.keytab.
I20260812 14:22:59.560362 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.60:38841
--webserver_interface=127.5.148.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.60.keytab
--principal=oryx/127.5.148.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:33907,127.5.148.61:35517,127.5.148.60:38841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:22:59.651409 13664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:59.651678 13664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:59.651719 13664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:59.652979 13664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:22:59.653059 13664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:59.653083 13664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:59.653105 13664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:22:59.653142 13664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:22:59.654832 13664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:33907,127.5.148.61:35517,127.5.148.60:38841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.60.keytab
--principal=oryx/127.5.148.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.60:38841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.148.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:59.655126 13664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:59.655371 13664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:59.660439 13672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:59.660446 13674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:59.660602 13664 server_base.cc:1034] running on GCE node
W20260812 14:22:59.660508 13670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:59.660907 13664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:59.661231 13664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:59.662434 13664 hybrid_clock.cc:648] HybridClock initialized: now 1786544579662433 us; error 42 us; skew 500 ppm
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544579, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:59.665400 13664 init.cc:377] Logged in from keytab as oryx/127.5.148.60@KRBTEST.COM (short username oryx)
I20260812 14:22:59.665923 13664 webserver.cc:460] Webserver started at http://127.5.148.60:37969/ using document root <none> and password file <none>
I20260812 14:22:59.666172 13664 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:59.666240 13664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:59.666389 13664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:22:59.667517 13664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/data/instance:
uuid: "a56acc7854aa44b0bcf78df32f9bbbe2"
format_stamp: "Formatted at 2026-08-12 14:22:59 on dist-test-slave-8w9v"
server_key: "2903d1b557a5f006181ed6861850048c"
server_key_iv: "748f0a5d2b933fcc0bf17786fbc5fab3"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.667889 13664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/wal/instance:
uuid: "a56acc7854aa44b0bcf78df32f9bbbe2"
format_stamp: "Formatted at 2026-08-12 14:22:59 on dist-test-slave-8w9v"
server_key: "2903d1b557a5f006181ed6861850048c"
server_key_iv: "748f0a5d2b933fcc0bf17786fbc5fab3"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.669417 13664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:22:59.670168 13681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.670348 13664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:22:59.670457 13664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/wal
uuid: "a56acc7854aa44b0bcf78df32f9bbbe2"
format_stamp: "Formatted at 2026-08-12 14:22:59 on dist-test-slave-8w9v"
server_key: "2903d1b557a5f006181ed6861850048c"
server_key_iv: "748f0a5d2b933fcc0bf17786fbc5fab3"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.670532 13664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:59.701719 13664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:59.704341 13664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:59.704519 13664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:59.709326 13664 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.60:38841
I20260812 14:22:59.709365 13738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.60:38841 every 8 connection(s)
I20260812 14:22:59.709769 13664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/data/info.pb
I20260812 14:22:59.710829 13740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:22:59.712484 13740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:22:59.717340 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 13664
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544579, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.60@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
I20260812 14:22:59.722797 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.713412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.5.148.62:33907)
0812 14:22:59.713471 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.713474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.713489 (+ 15us) server_negotiation.cc:407] Connection header received
0812 14:22:59.714062 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.714063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.714077 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.714146 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.714815 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.715319 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.716455 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.716704 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.716783 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.719685 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.719687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.719687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.719695 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.721345 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.721777 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.721779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.721779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.721847 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.722116 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.722117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.722118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.722246 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.722323 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.722613 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.722705 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:22:59.723007 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.713187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.0.0.1:55108)
0812 14:22:59.713367 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.713412 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.713936 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.714228 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.714232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.714501 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.714702 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.714707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.715423 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.715424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.716339 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.716342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.716793 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.716794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.716932 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.717633 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.717642 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.719582 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.721447 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.721454 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.721456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.721690 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.721939 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.721939 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.721940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.722045 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.722325 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.722333 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.722468 (+ 135us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.722571 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.722732 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.148.60@KRBTEST.COM
I20260812 14:22:59.723601 13740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:22:59.727042 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.717982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49354 (local address 127.5.148.60:38841)
0812 14:22:59.718162 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.718166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.718183 (+ 17us) server_negotiation.cc:407] Connection header received
0812 14:22:59.718236 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.718238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.718260 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.718361 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.719127 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.719912 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.720987 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.721282 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.721342 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.723902 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.723910 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.723910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.723918 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.725743 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.726153 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.726156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.726157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.726223 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.726473 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.726474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.726475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.726612 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.726812 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.726855 (+ 43us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.726929 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544579, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.60@KRBTEST.COM for oryx/127.5.148.61@KRBTEST.COM
I20260812 14:22:59.731034 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.723885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35517 (local address 127.0.0.1:53812)
0812 14:22:59.723922 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.723933 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.723968 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.724196 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.724197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.724343 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.724413 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.724415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.725340 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.725341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.725946 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.725949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.726404 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.726405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.726485 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.726868 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.726873 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.728263 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.730046 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.730047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.730047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.730201 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.730418 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.730419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.730419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.730482 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.730753 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.730754 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.730800 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.730883 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.730971 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260812 14:22:59.731055 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.724000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53812 (local address 127.5.148.61:35517)
0812 14:22:59.724050 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.724052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.724062 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:22:59.724088 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.724089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.724101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.724194 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.724512 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.725249 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.726031 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.726334 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.726384 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.728349 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.728350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.728350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.728357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.729971 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.730275 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.730276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.730276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.730331 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.730548 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.730548 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.730549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.730650 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.730702 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.730901 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.730993 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:22:59.731616 13740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544579, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.60@KRBTEST.COM for oryx/127.5.148.60@KRBTEST.COM
I20260812 14:22:59.738663 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.731942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38841 (local address 127.0.0.1:49364)
0812 14:22:59.732071 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.732083 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.732183 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.732353 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.732354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.732481 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.732514 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.732515 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.733216 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.733216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.733709 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.733710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.734222 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.734223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.734268 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.734678 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.734683 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.735920 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.737595 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.737597 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.737597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.737782 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.738004 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.738005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.738005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.738072 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.738358 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.738359 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.738402 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.738474 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.738566 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260812 14:22:59.738739 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.732082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.5.148.60:38841)
0812 14:22:59.732179 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.732180 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.732190 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:22:59.732250 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.732251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.732260 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.732337 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.732583 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.733118 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.733887 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.734133 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.734195 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.736016 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.736018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.736018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.736024 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.737499 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.737861 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.737862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.737863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.737917 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.738136 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.738137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.738137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.738243 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.738296 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.738507 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.738613 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":75}
I20260812 14:22:59.740391 13740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: Bootstrap starting.
I20260812 14:22:59.741292 13740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: Neither blocks nor log segments found. Creating new log.
I20260812 14:22:59.741616 13740 log.cc:824] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: Log is configured to *not* fsync() on all Append() calls
I20260812 14:22:59.742448 13740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: No bootstrap required, opened a new log
I20260812 14:22:59.743963 13740 raft_consensus.cc:348] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } }
I20260812 14:22:59.744191 13740 raft_consensus.cc:374] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:22:59.744246 13740 raft_consensus.cc:729] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a56acc7854aa44b0bcf78df32f9bbbe2, State: Initialized, Role: FOLLOWER
I20260812 14:22:59.744426 13740 consensus_queue.cc:260] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } }
I20260812 14:22:59.744815 13740 sys_catalog.cc:564] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 14:22:59.747157 13760 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:22:59.747267 13760 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:22:59.749007 13746 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } } }
I20260812 14:22:59.749101 13746 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:22:59.750782 13664 master_runner.cc:428] Detected that this master a56acc7854aa44b0bcf78df32f9bbbe2 is joining an existing cluster
I20260812 14:22:59.750852 13664 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.148.60:38841
I20260812 14:22:59.759347 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.752182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.5.148.60:38841)
0812 14:22:59.752224 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.752226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.752584 (+ 358us) server_negotiation.cc:407] Connection header received
0812 14:22:59.752721 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.752722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.752733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.752791 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.753232 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.753987 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.755069 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.755349 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.755387 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.756631 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.756633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.756633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.756640 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.758207 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.758549 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.758551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.758551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.758616 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.758792 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.758793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.758793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.758939 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.759014 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.759145 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.759262 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":17}
I20260812 14:22:59.759536 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.751954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38841 (local address 127.0.0.1:49370)
0812 14:22:59.752503 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.752515 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.752625 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.752832 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.752833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.753072 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.753145 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.753147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.754079 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.754079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.754976 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.754979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.755475 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.755476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.755537 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.756183 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.756189 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.756529 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.758292 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.758293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.758293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.758471 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.758685 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.758685 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.758686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.758735 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.759003 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.759004 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.759044 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.759106 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.759193 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":91,"threads_started":1}
I20260812 14:22:59.766227 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.752496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.0.0.1:55116)
0812 14:22:59.759806 (+ 7310us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.759818 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.759897 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.760177 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.760178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.760321 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.760377 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.760378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.761380 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.761380 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.761884 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.761886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.762379 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.762380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.762437 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.762933 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.762938 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.763230 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.765217 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.765218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.765219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.765417 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.765624 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.765625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.765625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.765691 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.765981 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.765982 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.766020 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.766076 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.766157 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7295,"thread_start_us":79,"threads_started":1}
I20260812 14:22:59.766309 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.752406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.5.148.62:33907)
0812 14:22:59.752456 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.752458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.759866 (+ 7408us) server_negotiation.cc:407] Connection header received
0812 14:22:59.759970 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.759971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.759986 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.760020 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.760463 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.761262 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.761977 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.762292 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.762367 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.763433 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.763434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.763435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.763460 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.765125 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.765489 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.765490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.765491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.765555 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.765753 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.765754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.765754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.765894 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.765975 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.766124 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.766207 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544578, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.62@KRBTEST.COM for oryx/127.5.148.60@KRBTEST.COM
I20260812 14:22:59.775663 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.768565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38841 (local address 127.0.0.1:49374)
0812 14:22:59.768620 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.768632 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.768673 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.768795 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.768796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.768960 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.768999 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.769001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.769621 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.769622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.770126 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.770127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.770491 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.770492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.770558 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.770973 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.770977 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.772484 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.774566 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.774567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.774568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.774781 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.775000 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.775000 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.775001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.775061 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.775335 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.775336 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.775379 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.775448 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.775557 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 14:22:59.776038 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.768635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49374 (local address 127.5.148.60:38841)
0812 14:22:59.768682 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.768684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.768694 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:22:59.768733 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.768734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.768744 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.768793 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.769085 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.769542 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.770210 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.770418 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.770493 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.772582 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.772583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.772583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.772591 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.774471 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.774858 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.774859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.774860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.774927 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.775125 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.775126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.775126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.775253 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.775310 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.775482 (+ 172us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.775604 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:22:59.776620 12884 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.148.60:38841: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2"
member_type: NON_VOTER
last_known_addr {
host: "127.5.148.60"
port: 38841
}
attrs {
promote: true
}
}
dest_uuid: "8bac19a5015e4f9085a7a1c559f4a309"
cas_config_opid_index: 5
I20260812 14:22:59.776773 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.752187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.5.148.61:35517)
0812 14:22:59.752238 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.752241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.753003 (+ 762us) server_negotiation.cc:407] Connection header received
0812 14:22:59.753148 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.753149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.753187 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.753220 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.753898 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.754804 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.755546 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.755820 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.755889 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.773858 (+ 17969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.773859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.773860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.773868 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.775290 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.775738 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.775740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.775740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.775809 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.776027 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.776028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.776029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.776349 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.776426 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.776588 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.776676 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:22:59.776922 12884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: NON_VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: true } }
I20260812 14:22:59.777323 13557 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a56acc7854aa44b0bcf78df32f9bbbe2 (127.5.148.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: NON_VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: true } } }
I20260812 14:22:59.777535 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.752255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35517 (local address 127.0.0.1:53814)
0812 14:22:59.752916 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.752927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.753032 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.753361 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.753362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.753648 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.753735 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.753738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.754896 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.754897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.755444 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.755445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.755912 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.755921 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.756002 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.772879 (+ 16877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.772885 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.773723 (+ 838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.775392 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.775393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.775394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.775646 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.775885 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.775886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.775886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.775938 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.776429 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.776430 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.776480 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.776560 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.777417 (+ 857us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":76,"threads_started":1}
I20260812 14:22:59.778424 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 12962
I20260812 14:22:59.778501 13557 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.148.60:38841
I20260812 14:22:59.778580 13557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: NON_VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: true } } }
I20260812 14:22:59.778664 13557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:59.782105 13710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:22:59.783123 13710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 FOLLOWER]: Refusing update from remote peer 8bac19a5015e4f9085a7a1c559f4a309: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 14:22:59.783986 13558 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: NON_VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:22:59.785521 13710 raft_consensus.cc:858] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: NON_VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: true } }
W20260812 14:22:59.785580 13710 raft_consensus.cc:1565] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: NON_VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: true } }
I20260812 14:22:59.786352 13710 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 36dc93bfb0624fe8af2565d5bd070107 (127.5.148.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } }
I20260812 14:22:59.787676 13746 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bac19a5015e4f9085a7a1c559f4a309. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } }
I20260812 14:22:59.787756 13746 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 14:22:59.787786 13746 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } }
I20260812 14:22:59.787813 13746 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 14:22:59.787835 13746 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } }
I20260812 14:22:59.787873 13746 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 14:22:59.790740 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.777540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.5.148.61:35517)
0812 14:22:59.777591 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.777592 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.778025 (+ 433us) server_negotiation.cc:407] Connection header received
0812 14:22:59.778123 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.778124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.778151 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.778204 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:22:59.778365 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.779065 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.789479 (+ 10414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.790163 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.790220 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.790295 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.790618 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.790662 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:22:59.790993 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.777448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35517 (local address 127.0.0.1:53830)
0812 14:22:59.777615 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.777625 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.778048 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.778256 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.778257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.778259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:22:59.778298 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.778300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.787953 (+ 9653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.787955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.789307 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.789312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.790362 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.790363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.790436 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.790464 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.790861 (+ 397us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.790912 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260812 14:22:59.791247 13615 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 LEARNER]: Refusing update from remote peer 8bac19a5015e4f9085a7a1c559f4a309: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 14:22:59.791450 13558 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 14:22:59.792402 13615 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a56acc7854aa44b0bcf78df32f9bbbe2 (127.5.148.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: NON_VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: true } } }
I20260812 14:22:59.793694 13647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bac19a5015e4f9085a7a1c559f4a309. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: NON_VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: true } } }
I20260812 14:22:59.793792 13647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: This master's current role is: LEARNER
I20260812 14:22:59.793730 13778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: NON_VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: true } } }
I20260812 14:22:59.793946 13558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: NON_VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: true } } }
I20260812 14:22:59.794036 13558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:59.794018 13778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: This master's current role is: LEARNER
I20260812 14:22:59.794302 13710 raft_consensus.cc:1467] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260812 14:22:59.794428 13710 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a56acc7854aa44b0bcf78df32f9bbbe2 (127.5.148.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: NON_VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: true } } }
I20260812 14:22:59.795320 13775 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: NON_VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: true } } }
I20260812 14:22:59.795457 13557 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309: attempting to promote NON_VOTER a56acc7854aa44b0bcf78df32f9bbbe2 to VOTER
I20260812 14:22:59.795486 13775 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: This master's current role is: LEARNER
I20260812 14:22:59.795662 13557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } }
I20260812 14:22:59.796229 13710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 LEARNER]: Refusing update from remote peer 8bac19a5015e4f9085a7a1c559f4a309: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 14:22:59.796458 13558 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 14:22:59.796697 13614 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 LEARNER]: Refusing update from remote peer 8bac19a5015e4f9085a7a1c559f4a309: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 14:22:59.796957 13558 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 14:22:59.797345 13779 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a56acc7854aa44b0bcf78df32f9bbbe2 (127.5.148.60) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.797511 13710 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a56acc7854aa44b0bcf78df32f9bbbe2 (127.5.148.60) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.798357 13775 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.798452 13775 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:22:59.798977 13557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.799072 13557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:59.798987 13614 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a56acc7854aa44b0bcf78df32f9bbbe2 (127.5.148.60) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.799249 13557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.799337 13557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:59.799516 13557 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309: attempting to promote NON_VOTER 36dc93bfb0624fe8af2565d5bd070107 to VOTER
I20260812 14:22:59.799638 13557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } }
I20260812 14:22:59.800531 13614 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 LEARNER]: Refusing update from remote peer 8bac19a5015e4f9085a7a1c559f4a309: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 14:22:59.800688 13771 mvcc.cc:205] Tried to move back new op lower bound from 7317686594286891008 to 7317686594205003776. Current Snapshot: MvccSnapshot[applied={T|T < 7317686594229039104}]
I20260812 14:22:59.800768 13558 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 14:22:59.801038 13710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 FOLLOWER]: Refusing update from remote peer 8bac19a5015e4f9085a7a1c559f4a309: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 14:22:59.801251 13558 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 14:22:59.801553 13614 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 36dc93bfb0624fe8af2565d5bd070107 (127.5.148.61) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.801808 13558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: NON_VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: true } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } } pending_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.801919 13558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:59.802075 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:43061
--local_ip_for_outbound_sockets=127.5.148.1
--tserver_master_addrs=127.5.148.62:33907
--webserver_port=40715
--webserver_interface=127.5.148.1
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.1.keytab
--principal=oryx/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.148.62:33907,127.5.148.61:35517
--tserver_master_addrs=127.5.148.62:33907,127.5.148.61:35517,127.5.148.60:38841 with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:22:59.802059 13779 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 36dc93bfb0624fe8af2565d5bd070107 (127.5.148.61) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.802430 13647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.802515 13647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:22:59.803078 13709 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 36dc93bfb0624fe8af2565d5bd070107 (127.5.148.61) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.803639 13557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.803738 13557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:59.803876 13664 master.cc:537] Master@127.5.148.60:38841 shutting down...
I20260812 14:22:59.803908 13557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.803979 13557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:59.804498 13773 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.804692 13773 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:22:59.804998 13778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.805079 13778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:22:59.805611 13779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.805718 13779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309 [sys.catalog]: This master's current role is: LEADER
I20260812 14:22:59.809561 13664 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:22:59.809684 13664 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:22:59.809728 13664 tablet_replica.cc:330] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: stopping tablet replica
I20260812 14:22:59.827435 13664 master.cc:552] Master@127.5.148.60:38841 shutdown complete.
I20260812 14:22:59.827518 13664 master_runner.cc:305] Clearing existing system tablet
I20260812 14:22:59.827668 13664 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:59.827732 13664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:59.828467 13664 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260812 14:22:59.829070 13786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.829249 13664 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:59.829312 13664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:22:59.829346 13664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/wal
uuid: "a56acc7854aa44b0bcf78df32f9bbbe2"
format_stamp: "Formatted at 2026-08-12 14:22:59 on dist-test-slave-8w9v"
server_key: "2903d1b557a5f006181ed6861850048c"
server_key_iv: "748f0a5d2b933fcc0bf17786fbc5fab3"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.830144 13664 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:22:59.831727 13664 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 14:22:59.831821 13664 log.cc:1196] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 14:22:59.832093 13664 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: Deleting consensus metadata
I20260812 14:22:59.832274 13664 master_runner.cc:315] Copying system tablet from 127.5.148.62:33907
I20260812 14:22:59.832950 13664 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: tablet copy: Beginning tablet copy session from remote peer at address 127.5.148.62:33907
I20260812 14:22:59.840052 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.833352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.5.148.62:33907)
0812 14:22:59.833404 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.833407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.833527 (+ 120us) server_negotiation.cc:407] Connection header received
0812 14:22:59.833610 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.833611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.833621 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.833681 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.834012 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.834517 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.835238 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.835451 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.835538 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.836779 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.836781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.836781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.836791 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.838605 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.839085 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.839086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.839087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.839163 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.839418 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.839419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.839419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.839559 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.839651 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.839843 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.839940 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:22:59.840073 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.833308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.0.0.1:55120)
0812 14:22:59.833445 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.833455 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.833542 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.833686 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.833687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.833874 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.833926 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.833928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.834605 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.834607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.835149 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.835151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.835540 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.835541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.835616 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.836154 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.836160 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.836559 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.838733 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.838743 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.838743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.838977 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.839254 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.839256 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.839256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.839337 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.839642 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.839643 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.839699 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.839810 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.839942 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260812 14:22:59.840950 12906 tablet_copy_service.cc:140] P 8bac19a5015e4f9085a7a1c559f4a309: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a56acc7854aa44b0bcf78df32f9bbbe2 ({username='oryx', principal='oryx/127.5.148.60@KRBTEST.COM'} at 127.0.0.1:55120)
I20260812 14:22:59.841028 12906 tablet_copy_service.cc:161] P 8bac19a5015e4f9085a7a1c559f4a309: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a56acc7854aa44b0bcf78df32f9bbbe2 at {username='oryx', principal='oryx/127.5.148.60@KRBTEST.COM'} at 127.0.0.1:55120: session id = a56acc7854aa44b0bcf78df32f9bbbe2-00000000000000000000000000000000
I20260812 14:22:59.841565 12906 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 14:22:59.842125 13664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 14:22:59.843277 13664 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: tablet copy: Starting download of 0 data blocks...
I20260812 14:22:59.843477 13664 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: tablet copy: Starting download of 1 WAL segments...
I20260812 14:22:59.844449 13664 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 14:22:59.845502 12906 tablet_copy_service.cc:342] P 8bac19a5015e4f9085a7a1c559f4a309: Request end of tablet copy session a56acc7854aa44b0bcf78df32f9bbbe2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.148.60@KRBTEST.COM'} at 127.0.0.1:55120
I20260812 14:22:59.845599 12906 tablet_copy_service.cc:433] P 8bac19a5015e4f9085a7a1c559f4a309: ending tablet copy session a56acc7854aa44b0bcf78df32f9bbbe2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a56acc7854aa44b0bcf78df32f9bbbe2
I20260812 14:22:59.848737 13664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:59.850029 13794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:59.850059 13798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:59.850569 13795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:59.850889 13664 server_base.cc:1034] running on GCE node
I20260812 14:22:59.851012 13664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:59.851186 13664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:59.852334 13664 hybrid_clock.cc:648] HybridClock initialized: now 1786544579852322 us; error 33 us; skew 500 ppm
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544579, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:59.853827 13664 init.cc:377] Logged in from keytab as oryx/127.5.148.60@KRBTEST.COM (short username oryx)
I20260812 14:22:59.854143 13664 webserver.cc:460] Webserver started at http://127.5.148.60:46319/ using document root <none> and password file <none>
I20260812 14:22:59.854270 13664 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:59.854308 13664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:59.854948 13664 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:22:59.855542 13805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.855701 13664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.855752 13664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/wal
uuid: "a56acc7854aa44b0bcf78df32f9bbbe2"
format_stamp: "Formatted at 2026-08-12 14:22:59 on dist-test-slave-8w9v"
server_key: "2903d1b557a5f006181ed6861850048c"
server_key_iv: "748f0a5d2b933fcc0bf17786fbc5fab3"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.855829 13664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:59.862388 13664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:59.864578 13664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:59.868003 13664 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.60:38841
I20260812 14:22:59.868623 13664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/master-2/data/info.pb
I20260812 14:22:59.869004 13865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.60:38841 every 8 connection(s)
I20260812 14:22:59.872862 13866 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:22:59.873353 13866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: Bootstrap starting.
I20260812 14:22:59.875558 13866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:22:59.875849 13866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: Bootstrap complete.
I20260812 14:22:59.876214 13866 raft_consensus.cc:348] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } }
I20260812 14:22:59.876338 13866 raft_consensus.cc:729] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a56acc7854aa44b0bcf78df32f9bbbe2, State: Initialized, Role: FOLLOWER
I20260812 14:22:59.876415 13866 consensus_queue.cc:260] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } }
I20260812 14:22:59.876641 13866 sys_catalog.cc:564] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:22:59.876633 13872 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:22:59.876929 13872 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:22:59.878423 13883 catalog_manager.cc:1259] Loaded cluster ID: a52194445a7e4e6ca42c123286c3fa53
I20260812 14:22:59.878530 13883 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: loading cluster ID for follower catalog manager: success
I20260812 14:22:59.879779 13883 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: acquiring CA information for follower catalog manager: success
I20260812 14:22:59.880087 13883 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 14:22:59.931083 13784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:22:59.931380 13784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:22:59.931414 13784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:22:59.933213 13784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:22:59.933277 13784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:22:59.933318 13784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:22:59.935606 13784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.1.keytab
--principal=oryx/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:43061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=40715
--tserver_master_addrs=127.5.148.62:33907,127.5.148.61:35517,127.5.148.60:38841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:22:59.935889 13784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:22:59.936219 13784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:22:59.942045 13890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:59.942056 13892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:22:59.942061 13888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:22:59.942494 13784 server_base.cc:1034] running on GCE node
I20260812 14:22:59.942704 13784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:22:59.942958 13784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:22:59.944161 13784 hybrid_clock.cc:648] HybridClock initialized: now 1786544579944144 us; error 29 us; skew 500 ppm
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544579, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:22:59.947130 13784 init.cc:377] Logged in from keytab as oryx/127.5.148.1@KRBTEST.COM (short username oryx)
I20260812 14:22:59.947618 13784 webserver.cc:460] Webserver started at http://127.5.148.1:40715/ using document root <none> and password file <none>
I20260812 14:22:59.947872 13784 fs_manager.cc:359] Metadata directory not provided
I20260812 14:22:59.947937 13784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:22:59.949469 13784 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:59.950140 13900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.950341 13784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:22:59.950444 13784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054"
format_stamp: "Formatted at 2026-08-12 14:22:58 on dist-test-slave-8w9v"
server_key: "1e4a25fa8a34686aef970433de994164"
server_key_iv: "3b9a1a1b7035b4a1d0f51e40509a92b5"
server_key_version: "encryptionkey@0"
I20260812 14:22:59.950824 13784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:22:59.970009 13784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:22:59.972592 13784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:22:59.972769 13784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:22:59.973029 13784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:22:59.973376 13784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:22:59.973433 13784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.973482 13784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:22:59.973513 13784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:22:59.980254 13784 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:43061
I20260812 14:22:59.980343 14026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:43061 every 8 connection(s)
I20260812 14:22:59.980705 13784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:22:59.983690 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 13784
I20260812 14:22:59.983848 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 13126
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544579, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544579, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.60@KRBTEST.COM
I20260812 14:22:59.992666 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.981517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:47499 (local address 127.5.148.62:33907)
0812 14:22:59.981616 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.981619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.981977 (+ 358us) server_negotiation.cc:407] Connection header received
0812 14:22:59.982531 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.982533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.982547 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.982603 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.983516 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.984219 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.985554 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.985811 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.985859 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.989428 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.989429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.989430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.989438 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.991067 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.991517 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.991518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.991519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.991584 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:22:59.991882 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:22:59.991883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:22:59.991883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:22:59.992014 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:22:59.992095 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:22:59.992444 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 14:22:59.992531 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 14:22:59.993252 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.981499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.1:47499)
0812 14:22:59.981899 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.981909 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.982430 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.982679 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.982683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.983147 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.983416 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.983421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.984335 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.984336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.985462 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.985464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.985942 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.985943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.986079 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.986903 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.986916 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.989318 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:22:59.991176 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.991185 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.991188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.991425 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.991677 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:22:59.991678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:22:59.991678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:22:59.991805 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:22:59.992086 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:22:59.992094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:22:59.992296 (+ 202us) client_negotiation.cc:769] Sending connection context
0812 14:22:59.992788 (+ 492us) client_negotiation.cc:240] Negotiation successful
0812 14:22:59.992982 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":43,"thread_start_us":176,"threads_started":1}
I20260812 14:22:59.993913 14028 heartbeater.cc:347] Connected to a master server at 127.5.148.62:33907
I20260812 14:22:59.994010 14028 heartbeater.cc:464] Registering TS with master...
I20260812 14:22:59.994196 14028 heartbeater.cc:511] Master 127.5.148.62:33907 requested a full tablet report, sending...
I20260812 14:22:59.994654 12884 ts_manager.cc:194] Re-registered known tserver with Master: 8bbd51a2cc8c4fe6af24d4ffda7f4054 (127.5.148.1:43061)
I20260812 14:22:59.995098 12884 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:47499
Aug 12 14:22:59 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544579, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.61@KRBTEST.COM
I20260812 14:23:00.001566 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.981525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:57455 (local address 127.5.148.60:38841)
0812 14:22:59.981645 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.981647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.981862 (+ 215us) server_negotiation.cc:407] Connection header received
0812 14:22:59.982551 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.982552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.982566 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.982607 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.983537 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.984708 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.986060 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.986503 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.986555 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.997728 (+ 11173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:22:59.997730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:22:59.997730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:22:59.997741 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:22:59.999986 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.000505 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.000507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.000507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.000585 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.000837 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.000838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.000839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.001005 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:00.001083 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.001292 (+ 209us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.001412 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260812 14:23:00.002215 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.981498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38841 (local address 127.5.148.1:57455)
0812 14:22:59.981723 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.981777 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.982418 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.982679 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.982685 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.983168 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.983416 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.983423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.984827 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.984828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.985936 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.985939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.986651 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.986653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.986743 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.987174 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.987180 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:22:59.997617 (+ 10437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:00.000189 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.000190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.000191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.000416 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.000674 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.000676 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.000676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.000745 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.001090 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:00.001092 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:00.001148 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 14:23:00.001716 (+ 568us) client_negotiation.cc:240] Negotiation successful
0812 14:23:00.002126 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":32,"thread_start_us":144,"threads_started":1}
I20260812 14:23:00.002424 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:38119
--local_ip_for_outbound_sockets=127.5.148.2
--tserver_master_addrs=127.5.148.62:33907
--webserver_port=46463
--webserver_interface=127.5.148.2
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.2.keytab
--principal=oryx/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.148.62:33907,127.5.148.61:35517
--tserver_master_addrs=127.5.148.62:33907,127.5.148.61:35517,127.5.148.60:38841 with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:23:00.004547 14027 heartbeater.cc:347] Connected to a master server at 127.5.148.60:38841
I20260812 14:23:00.004612 14027 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:00.004711 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.981902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:44277 (local address 127.5.148.61:35517)
0812 14:22:59.981957 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:22:59.981959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:22:59.982081 (+ 122us) server_negotiation.cc:407] Connection header received
0812 14:22:59.982550 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:22:59.982551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:22:59.982565 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:22:59.982601 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:22:59.996784 (+ 14183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.997318 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.998782 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.999101 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.999168 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.001521 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:00.001523 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:00.001523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:00.001531 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:00.003388 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.003712 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.003713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.003713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.003779 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.003954 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.003955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.003955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.004135 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:00.004226 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.004482 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.004589 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:23:00.004734 14027 heartbeater.cc:511] Master 127.5.148.60:38841 requested a full tablet report, sending...
I20260812 14:23:00.005079 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:22:59.981714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35517 (local address 127.5.148.1:44277)
0812 14:22:59.982003 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0812 14:22:59.982013 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:22:59.982429 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:22:59.983099 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:22:59.983100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:22:59.983274 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:22:59.983416 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.983423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.997754 (+ 14331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.997755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.998571 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:22:59.998575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:22:59.999254 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:22:59.999254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:22:59.999327 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:22:59.999749 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:22:59.999755 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:00.001429 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:00.003473 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.003474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.003475 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.003636 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.003845 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.003846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.003846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.003896 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.004289 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:00.004291 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:00.004350 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 14:23:00.004427 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 14:23:00.004562 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":4,"thread_start_us":75,"threads_started":1}
I20260812 14:23:00.005198 13824 ts_manager.cc:194] Registered new tserver with Master: 8bbd51a2cc8c4fe6af24d4ffda7f4054 (127.5.148.1:43061)
I20260812 14:23:00.005517 14029 heartbeater.cc:347] Connected to a master server at 127.5.148.61:35517
I20260812 14:23:00.005584 14029 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:00.005640 14029 heartbeater.cc:511] Master 127.5.148.61:35517 requested a full tablet report, sending...
I20260812 14:23:00.006068 13601 ts_manager.cc:194] Registered new tserver with Master: 8bbd51a2cc8c4fe6af24d4ffda7f4054 (127.5.148.1:43061)
W20260812 14:23:00.090353 14036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:00.090596 14036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:00.090621 14036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:00.091758 14036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:00.091796 14036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:00.091822 14036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:23:00.093377 14036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.2.keytab
--principal=oryx/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:38119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=46463
--tserver_master_addrs=127.5.148.62:33907,127.5.148.61:35517,127.5.148.60:38841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:00.093583 14036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:00.093767 14036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:00.099215 14042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:00.099220 14045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:00.099267 14043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:00.099306 14036 server_base.cc:1034] running on GCE node
I20260812 14:23:00.099695 14036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:00.099958 14036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:00.101125 14036 hybrid_clock.cc:648] HybridClock initialized: now 1786544580101114 us; error 43 us; skew 500 ppm
Aug 12 14:23:00 dist-test-slave-8w9v krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544580, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:00.110628 14036 init.cc:377] Logged in from keytab as oryx/127.5.148.2@KRBTEST.COM (short username oryx)
I20260812 14:23:00.111167 14036 webserver.cc:460] Webserver started at http://127.5.148.2:46463/ using document root <none> and password file <none>
I20260812 14:23:00.111377 14036 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:00.111435 14036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:00.112913 14036 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:00.113526 14054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:00.113704 14036 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:00.113797 14036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "dec94ea6b5974186b3f220955b548336"
format_stamp: "Formatted at 2026-08-12 14:22:58 on dist-test-slave-8w9v"
server_key: "a7db0a781a7e405463c7e88fb594950c"
server_key_iv: "b0203f2906003a6145366be139b5eb9d"
server_key_version: "encryptionkey@0"
I20260812 14:23:00.114151 14036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:00.128311 14036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:00.130494 14036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:00.130652 14036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:00.130923 14036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:00.131268 14036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:00.131306 14036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:00.131330 14036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:00.131345 14036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:00.137504 14036 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:38119
I20260812 14:23:00.137555 14187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:38119 every 8 connection(s)
I20260812 14:23:00.137917 14036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:23:00.139451 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 14036
I20260812 14:23:00.139616 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 13292
Aug 12 14:23:00 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544580, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
I20260812 14:23:00.148839 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:39507
--local_ip_for_outbound_sockets=127.5.148.3
--tserver_master_addrs=127.5.148.62:33907
--webserver_port=37905
--webserver_interface=127.5.148.3
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.3.keytab
--principal=oryx/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.148.62:33907,127.5.148.61:35517
--tserver_master_addrs=127.5.148.62:33907,127.5.148.61:35517,127.5.148.60:38841 with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:23:00 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544580, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.60@KRBTEST.COM
Aug 12 14:23:00 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544580, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.61@KRBTEST.COM
I20260812 14:23:00.152443 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.139003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:43615 (local address 127.5.148.62:33907)
0812 14:23:00.139050 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.139052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.139095 (+ 43us) server_negotiation.cc:407] Connection header received
0812 14:23:00.140581 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.140583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.140597 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.140625 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:00.141732 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.142691 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.144067 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.144385 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.144458 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.148359 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:00.148361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:00.148361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:00.148370 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:00.150187 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.150726 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.150729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.150729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.150791 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.151305 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.151306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.151306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.151475 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:00.151552 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.152243 (+ 691us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.152337 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:23:00.153095 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.138700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.2:43615)
0812 14:23:00.139024 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:00.139036 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:00.140460 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:00.140795 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:00.140798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:00.141196 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:00.141594 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.141603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.142817 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.142818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.143965 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.143967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.144481 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.144482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.144647 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.146096 (+ 1449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:00.146107 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:00.148248 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:00.150282 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.150291 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.150294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.150635 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.150868 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.150869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.150869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.151030 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.151541 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:00.151549 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:00.151733 (+ 184us) client_negotiation.cc:769] Sending connection context
0812 14:23:00.152595 (+ 862us) client_negotiation.cc:240] Negotiation successful
0812 14:23:00.152782 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"spinlock_wait_cycles":7936,"thread_start_us":101,"threads_started":1}
I20260812 14:23:00.153577 14189 heartbeater.cc:347] Connected to a master server at 127.5.148.62:33907
I20260812 14:23:00.153703 14189 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:00.153973 14189 heartbeater.cc:511] Master 127.5.148.62:33907 requested a full tablet report, sending...
I20260812 14:23:00.154134 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.139329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:60237 (local address 127.5.148.61:35517)
0812 14:23:00.139388 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.139390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.139546 (+ 156us) server_negotiation.cc:407] Connection header received
0812 14:23:00.144213 (+ 4667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.144215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.144231 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.144274 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:00.145109 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.145906 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.147220 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.147541 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.147588 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.150098 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:00.150100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:00.150100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:00.150108 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:00.151933 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.152370 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.152371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.152372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.152437 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.153024 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.153025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.153026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.153167 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:00.153226 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.153915 (+ 689us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.154008 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:23:00.154300 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.138696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:39403 (local address 127.5.148.60:38841)
0812 14:23:00.138964 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.138967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.139029 (+ 62us) server_negotiation.cc:407] Connection header received
0812 14:23:00.144239 (+ 5210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.144240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.144259 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.144289 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:00.144756 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.145608 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.146324 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.146633 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.146698 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.150949 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:00.150950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:00.150951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:00.150958 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:00.153017 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.153405 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.153406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.153406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.153488 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.153724 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.153725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.153725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.153859 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:00.153905 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.154110 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.154202 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260812 14:23:00.154359 12884 ts_manager.cc:194] Re-registered known tserver with Master: dec94ea6b5974186b3f220955b548336 (127.5.148.2:38119)
I20260812 14:23:00.154712 12884 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:43615
I20260812 14:23:00.155010 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.138689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38841 (local address 127.5.148.2:39403)
0812 14:23:00.138882 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:00.138937 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:00.144150 (+ 5213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:00.144380 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:00.144381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:00.144589 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:00.144664 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.144666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.145700 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.145702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.146233 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.146234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.146753 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.146754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.146820 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.147314 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:00.147319 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:00.150846 (+ 3527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:00.153110 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.153112 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.153112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.153321 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.153567 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.153568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.153569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.153645 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.153954 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:00.153954 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:00.153994 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:23:00.154375 (+ 381us) client_negotiation.cc:240] Negotiation successful
0812 14:23:00.154864 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":347,"thread_start_us":95,"threads_started":1}
I20260812 14:23:00.155022 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.139270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35517 (local address 127.5.148.2:60237)
0812 14:23:00.139468 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:00.139477 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:00.144077 (+ 4600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:00.144791 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:00.144792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:00.144952 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:00.145013 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.145015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.146369 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.146370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.147106 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.147109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.147670 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.147670 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.147740 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.148196 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:00.148202 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:00.150008 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:00.152027 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.152028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.152029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.152272 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.152850 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.152851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.152851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.152945 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.153509 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:00.153510 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:00.153559 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 14:23:00.153636 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 14:23:00.154736 (+ 1100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
I20260812 14:23:00.155362 14188 heartbeater.cc:347] Connected to a master server at 127.5.148.60:38841
I20260812 14:23:00.155412 14188 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:00.155504 14188 heartbeater.cc:511] Master 127.5.148.60:38841 requested a full tablet report, sending...
I20260812 14:23:00.155877 13824 ts_manager.cc:194] Registered new tserver with Master: dec94ea6b5974186b3f220955b548336 (127.5.148.2:38119)
I20260812 14:23:00.156311 14190 heartbeater.cc:347] Connected to a master server at 127.5.148.61:35517
I20260812 14:23:00.156363 14190 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:00.156450 14190 heartbeater.cc:511] Master 127.5.148.61:35517 requested a full tablet report, sending...
I20260812 14:23:00.156786 13601 ts_manager.cc:194] Registered new tserver with Master: dec94ea6b5974186b3f220955b548336 (127.5.148.2:38119)
W20260812 14:23:00.238229 14196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:00.238466 14196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:00.238494 14196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:00.239638 14196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:00.239679 14196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:00.239706 14196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:23:00.241247 14196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.3.keytab
--principal=oryx/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:39507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=37905
--tserver_master_addrs=127.5.148.62:33907,127.5.148.61:35517,127.5.148.60:38841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:00.241458 14196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:00.241652 14196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:00.247033 14202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:00.247051 14203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:00.247033 14206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:00.247443 14196 server_base.cc:1034] running on GCE node
I20260812 14:23:00.247709 14196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:00.248004 14196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:00.249181 14196 hybrid_clock.cc:648] HybridClock initialized: now 1786544580249172 us; error 39 us; skew 500 ppm
Aug 12 14:23:00 dist-test-slave-8w9v krb5kdc[12829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544580, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:00.252259 14196 init.cc:377] Logged in from keytab as oryx/127.5.148.3@KRBTEST.COM (short username oryx)
I20260812 14:23:00.252775 14196 webserver.cc:460] Webserver started at http://127.5.148.3:37905/ using document root <none> and password file <none>
I20260812 14:23:00.253052 14196 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:00.253131 14196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:00.254585 14196 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:23:00.255290 14214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:00.255511 14196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:00.255606 14196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "62540953877147abab1c3c4206ff0bc7"
format_stamp: "Formatted at 2026-08-12 14:22:59 on dist-test-slave-8w9v"
server_key: "b9601f574aa4fb0791b2aa710b987231"
server_key_iv: "06ad6210c262e66eede751924aec8e36"
server_key_version: "encryptionkey@0"
I20260812 14:23:00.255975 14196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:00.273546 14196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:00.276185 14196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:00.276314 14196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:00.276535 14196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:00.276893 14196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:00.276932 14196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:00.276957 14196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:00.277002 14196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:00.283243 14196 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:39507
I20260812 14:23:00.283324 14342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:39507 every 8 connection(s)
I20260812 14:23:00.283725 14196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:23:00.286008 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 14196
Aug 12 14:23:00 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544580, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.61@KRBTEST.COM
Aug 12 14:23:00 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544580, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.60@KRBTEST.COM
Aug 12 14:23:00 dist-test-slave-8w9v krb5kdc[12829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544580, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
I20260812 14:23:00.297708 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.284889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:47659 (local address 127.5.148.60:38841)
0812 14:23:00.284940 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.284942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.285030 (+ 88us) server_negotiation.cc:407] Connection header received
0812 14:23:00.286174 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.286175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.286190 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.286224 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:00.287419 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.288231 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.290008 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.290299 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.290340 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.294106 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:00.294107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:00.294108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:00.294115 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:00.295848 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.296458 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.296460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.296460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.296525 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.296846 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.296847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.296848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.297001 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:00.297048 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.297478 (+ 430us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.297574 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:23:00.298564 14348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.284553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38841 (local address 127.5.148.3:47659)
0812 14:23:00.284968 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:00.284977 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:00.286040 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:00.286436 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:00.286440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:00.287048 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:00.287294 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.287300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.288383 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.288384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.289897 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.289900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.290434 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.290434 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.290559 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.292179 (+ 1620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:00.292185 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:00.294021 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:00.295931 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.295940 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.295942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.296341 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.296604 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.296605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.296605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.296756 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.297099 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:00.297107 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:00.297289 (+ 182us) client_negotiation.cc:769] Sending connection context
0812 14:23:00.297821 (+ 532us) client_negotiation.cc:240] Negotiation successful
0812 14:23:00.298024 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":132,"thread_start_us":118,"threads_started":1}
I20260812 14:23:00.299222 14343 heartbeater.cc:347] Connected to a master server at 127.5.148.60:38841
I20260812 14:23:00.299368 14343 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:00.299635 14343 heartbeater.cc:511] Master 127.5.148.60:38841 requested a full tablet report, sending...
I20260812 14:23:00.300143 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.284572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:53893 (local address 127.5.148.62:33907)
0812 14:23:00.284629 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.284631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.285074 (+ 443us) server_negotiation.cc:407] Connection header received
0812 14:23:00.286175 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.286176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.286192 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.286227 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:00.287507 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.288262 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.289989 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.290276 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.290316 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.296291 (+ 5975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:00.296294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:00.296295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:00.296303 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:00.298013 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.298691 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.298692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.298693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.298751 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.299406 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.299407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.299407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.299561 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:00.299608 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.299934 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.300012 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:23:00.300184 13824 ts_manager.cc:194] Registered new tserver with Master: 62540953877147abab1c3c4206ff0bc7 (127.5.148.3:39507)
I20260812 14:23:00.300797 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.284644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.3:53893)
0812 14:23:00.285005 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:00.285016 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:00.286053 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:00.286436 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:00.286440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:00.287280 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:00.287393 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.287396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.288394 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.288395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.289900 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.289903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.290416 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.290416 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.290559 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.292772 (+ 2213us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:00.292778 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:00.296190 (+ 3412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:00.298340 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.298354 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.298354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.298591 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.299162 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.299163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.299163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.299226 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.299747 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:00.299749 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:00.299803 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 14:23:00.300474 (+ 671us) client_negotiation.cc:240] Negotiation successful
0812 14:23:00.300692 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":107,"thread_start_us":74,"threads_started":1}
I20260812 14:23:00.300911 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.287639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.5.148.61:35517)
0812 14:23:00.288612 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.288614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.288621 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:23:00.288641 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.288642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.288652 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.288742 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:00.291272 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.292025 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.293665 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.294012 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.294086 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.297247 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:00.297248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:00.297248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:00.297256 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:00.299168 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.299705 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.299705 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.299706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.299765 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.300021 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.300022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.300022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.300182 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:00.300608 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.300669 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.300769 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":54,"threads_started":1}
I20260812 14:23:00.301004 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.284909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:37081 (local address 127.5.148.61:35517)
0812 14:23:00.284981 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.284982 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.284994 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:23:00.286174 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.286175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.286189 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.286224 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:00.287429 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.288236 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.290106 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.290402 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.290441 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.295779 (+ 5338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:00.295781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:00.295781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:00.295789 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:00.298285 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.299480 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.299481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.299482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.299537 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.300319 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.300320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.300321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.300448 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:00.300492 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.300706 (+ 214us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.300834 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":47}
I20260812 14:23:00.301566 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.284553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35517 (local address 127.5.148.3:37081)
0812 14:23:00.284752 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:00.284798 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:00.286029 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:00.286436 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:00.286440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:00.286999 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:00.287294 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.287300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.288383 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.288384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.290020 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.290022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.290515 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.290515 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.290573 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.291527 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:00.291537 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:00.295674 (+ 4137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:00.298377 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.298378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.298378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.299223 (+ 845us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.300157 (+ 934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:00.300158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:00.300158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:00.300226 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:00.300539 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:00.300540 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:00.300580 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:23:00.300738 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 14:23:00.301453 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":111,"thread_start_us":115,"threads_started":1}
I20260812 14:23:00.301664 14344 heartbeater.cc:347] Connected to a master server at 127.5.148.62:33907
I20260812 14:23:00.301704 14344 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:00.301771 14344 heartbeater.cc:511] Master 127.5.148.62:33907 requested a full tablet report, sending...
I20260812 14:23:00.301890 14345 heartbeater.cc:347] Connected to a master server at 127.5.148.61:35517
I20260812 14:23:00.301924 14345 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:00.302013 14345 heartbeater.cc:511] Master 127.5.148.61:35517 requested a full tablet report, sending...
I20260812 14:23:00.302199 12884 ts_manager.cc:194] Re-registered known tserver with Master: 62540953877147abab1c3c4206ff0bc7 (127.5.148.3:39507)
I20260812 14:23:00.302404 13601 ts_manager.cc:194] Registered new tserver with Master: 62540953877147abab1c3c4206ff0bc7 (127.5.148.3:39507)
I20260812 14:23:00.302685 12884 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:53893
I20260812 14:23:00.303262 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.287866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.5.148.62:33907)
0812 14:23:00.288101 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.288103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.288850 (+ 747us) server_negotiation.cc:407] Connection header received
0812 14:23:00.288875 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.288876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.288889 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.288963 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:00.289823 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.290562 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.292261 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.292590 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.292665 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.295384 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:00.295385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:00.295386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:00.295393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:00.301449 (+ 6056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.301857 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.301859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.301859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.301934 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.302182 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.302183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.302183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.302609 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:00.302704 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.303059 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.303153 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":50,"threads_started":1}
I20260812 14:23:00.303758 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.287365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.5.148.60:38841)
0812 14:23:00.288441 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.288443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.288450 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:23:00.288471 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.288472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.288484 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.288543 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:00.291077 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.291839 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.292947 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.293265 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.293335 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.295467 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:00.295468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:00.295468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:00.295476 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:00.302079 (+ 6603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.302491 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.302492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.302492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.302560 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.302820 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.302821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.302822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.302978 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:00.303072 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.303271 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.303672 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":56,"threads_started":1}
I20260812 14:23:00.304908 12884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:23:00.306650 12884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:23:00.311571 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.307809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38841 (local address 127.0.0.1:49390)
0812 14:23:00.307942 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:00.307953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:00.308035 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:00.308331 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:00.308331 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:00.308333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:00.308378 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.308379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.309399 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.309400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.310437 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.310441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.311305 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.311306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.311376 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.311382 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 14:23:00.311416 (+ 34us) client_negotiation.cc:240] Negotiation successful
0812 14:23:00.311465 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260812 14:23:00.312064 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.308014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.5.148.60:38841)
0812 14:23:00.308081 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.308083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.308094 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:23:00.308163 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.308164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.308193 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.308327 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:00.308442 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.309243 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.310574 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.311197 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.311815 (+ 618us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.311918 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.311943 (+ 25us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.311998 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:23:00.312650 13872 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bac19a5015e4f9085a7a1c559f4a309. Latest consensus state: current_term: 1 leader_uuid: "8bac19a5015e4f9085a7a1c559f4a309" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:23:00.312798 13872 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:23:00.316763 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.311462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:43061 (local address 127.0.0.1:55492)
0812 14:23:00.311658 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:00.311669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:00.311707 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:00.312692 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:00.312693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:00.312695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:00.312732 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.312733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.314275 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.314276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.315257 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.315259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.316506 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.316507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.316585 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.316590 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:23:00.316642 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 14:23:00.316685 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
I20260812 14:23:00.316862 14373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.312042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55492 (local address 127.5.148.1:43061)
0812 14:23:00.312478 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.312485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.312497 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:23:00.312545 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.312547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.312566 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.312988 (+ 422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:00.313060 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.314171 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.315355 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.316395 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.316503 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.316644 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.316676 (+ 32us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.316721 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":333,"threads_started":1}
I20260812 14:23:00.317673 13954 tablet_service.cc:1467] Processing CreateTablet for tablet 01133b830f4d482a93150990395c6e66 (DEFAULT_TABLE table=test-table [id=29de299c64ca4e1a96c616a6a2248faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:00.318084 13954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01133b830f4d482a93150990395c6e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:00.318639 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.312366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:39507 (local address 127.0.0.1:41720)
0812 14:23:00.314314 (+ 1948us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:00.314326 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:00.314367 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:00.314656 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:00.314657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:00.314660 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:00.314698 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.314700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.315835 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.315836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.317173 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.317176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.318307 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.318308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.318432 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.318439 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 14:23:00.318516 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 14:23:00.318560 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1924}
I20260812 14:23:00.318688 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.311598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.5.148.3:39507)
0812 14:23:00.314483 (+ 2885us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.314488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.314496 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:23:00.314536 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.314538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.314552 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.314619 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:00.314804 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.315739 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.317261 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.318197 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.318277 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.318336 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.318550 (+ 214us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.318597 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2847,"thread_start_us":569,"threads_started":1}
I20260812 14:23:00.319088 14369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.311638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:38119 (local address 127.0.0.1:40590)
0812 14:23:00.312200 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:00.312213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:00.312877 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:00.314896 (+ 2019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:00.314897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:00.314900 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:00.314947 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.314949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.316282 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.316283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.317493 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.317497 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.318821 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.318822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.318902 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.318906 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:23:00.318934 (+ 28us) client_negotiation.cc:240] Negotiation successful
0812 14:23:00.319004 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":145,"threads_started":1}
I20260812 14:23:00.319545 14372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.312209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40590 (local address 127.5.148.2:38119)
0812 14:23:00.314737 (+ 2528us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.314741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.314749 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:23:00.314786 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.314787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.314800 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.314899 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:00.315014 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.316149 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.317758 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.318684 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.319297 (+ 613us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.319390 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.319418 (+ 28us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.319463 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2494,"thread_start_us":85,"threads_started":1}
I20260812 14:23:00.319645 14268 tablet_service.cc:1467] Processing CreateTablet for tablet 01133b830f4d482a93150990395c6e66 (DEFAULT_TABLE table=test-table [id=29de299c64ca4e1a96c616a6a2248faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:00.320014 14268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01133b830f4d482a93150990395c6e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:00.320222 14106 tablet_service.cc:1467] Processing CreateTablet for tablet 01133b830f4d482a93150990395c6e66 (DEFAULT_TABLE table=test-table [id=29de299c64ca4e1a96c616a6a2248faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:00.320542 14106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01133b830f4d482a93150990395c6e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:00.322155 14378 tablet_bootstrap.cc:492] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054: Bootstrap starting.
I20260812 14:23:00.322934 14378 tablet_bootstrap.cc:654] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:00.323232 14379 tablet_bootstrap.cc:492] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7: Bootstrap starting.
I20260812 14:23:00.323326 14378 log.cc:824] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054: Log is configured to *not* fsync() on all Append() calls
I20260812 14:23:00.323901 14379 tablet_bootstrap.cc:654] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:00.324271 14381 tablet_bootstrap.cc:492] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336: Bootstrap starting.
I20260812 14:23:00.324260 14379 log.cc:824] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7: Log is configured to *not* fsync() on all Append() calls
I20260812 14:23:00.324365 14378 tablet_bootstrap.cc:492] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054: No bootstrap required, opened a new log
I20260812 14:23:00.324443 14378 ts_tablet_manager.cc:1397] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:23:00.324896 14381 tablet_bootstrap.cc:654] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:00.325176 14379 tablet_bootstrap.cc:492] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7: No bootstrap required, opened a new log
I20260812 14:23:00.325212 14381 log.cc:824] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336: Log is configured to *not* fsync() on all Append() calls
I20260812 14:23:00.325249 14379 ts_tablet_manager.cc:1397] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:23:00.326098 14381 tablet_bootstrap.cc:492] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336: No bootstrap required, opened a new log
I20260812 14:23:00.326234 14381 ts_tablet_manager.cc:1397] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:23:00.327234 14379 raft_consensus.cc:348] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } }
I20260812 14:23:00.327376 14379 raft_consensus.cc:374] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:00.327399 14379 raft_consensus.cc:729] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62540953877147abab1c3c4206ff0bc7, State: Initialized, Role: FOLLOWER
I20260812 14:23:00.327538 14379 consensus_queue.cc:260] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } }
I20260812 14:23:00.327564 14378 raft_consensus.cc:348] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } }
I20260812 14:23:00.327744 14378 raft_consensus.cc:374] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:00.327787 14378 raft_consensus.cc:729] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bbd51a2cc8c4fe6af24d4ffda7f4054, State: Initialized, Role: FOLLOWER
I20260812 14:23:00.327860 14379 ts_tablet_manager.cc:1428] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:23:00.327986 14344 heartbeater.cc:503] Master 127.5.148.62:33907 was elected leader, sending a full tablet report...
I20260812 14:23:00.327942 14378 consensus_queue.cc:260] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } }
I20260812 14:23:00.328625 14028 heartbeater.cc:503] Master 127.5.148.62:33907 was elected leader, sending a full tablet report...
I20260812 14:23:00.328606 14378 ts_tablet_manager.cc:1428] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260812 14:23:00.330130 14381 raft_consensus.cc:348] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } }
I20260812 14:23:00.330312 14381 raft_consensus.cc:374] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:00.330371 14381 raft_consensus.cc:729] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec94ea6b5974186b3f220955b548336, State: Initialized, Role: FOLLOWER
I20260812 14:23:00.330528 14381 consensus_queue.cc:260] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } }
I20260812 14:23:00.330850 14189 heartbeater.cc:503] Master 127.5.148.62:33907 was elected leader, sending a full tablet report...
I20260812 14:23:00.330853 14381 ts_tablet_manager.cc:1428] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260812 14:23:00.388618 14191 tablet.cc:2369] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:23:00.481755 14030 tablet.cc:2369] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:23:00.515631 14388 raft_consensus.cc:482] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:23:00.515758 14388 raft_consensus.cc:504] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } }
I20260812 14:23:00.516099 14388 leader_election.cc:288] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bbd51a2cc8c4fe6af24d4ffda7f4054 (127.5.148.1:43061), 62540953877147abab1c3c4206ff0bc7 (127.5.148.3:39507)
I20260812 14:23:00.519635 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.516331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:43061 (local address 127.5.148.2:41783)
0812 14:23:00.516385 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:00.516398 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:00.516463 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:00.516662 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:00.516663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:00.516666 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:00.516781 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.516783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.517503 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.517504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.518562 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.518566 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.519375 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.519376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.519457 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.519460 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:23:00.519518 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 14:23:00.519548 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":29}
I20260812 14:23:00.519660 14373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.516436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:41783 (local address 127.5.148.1:43061)
0812 14:23:00.516486 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.516488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.516502 (+ 14us) server_negotiation.cc:407] Connection header received
0812 14:23:00.516534 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.516535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.516547 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.516640 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:00.516844 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.517396 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.518686 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.519298 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.519368 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.519379 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.519534 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.519585 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:23:00.519757 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.516437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:49037 (local address 127.5.148.3:39507)
0812 14:23:00.516571 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.516573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.516582 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:23:00.516620 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.516621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.516632 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.516703 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:00.516887 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.517496 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.518667 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.519353 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.519428 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.519439 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.519632 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.519682 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260812 14:23:00.519763 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.516337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:39507 (local address 127.5.148.2:49037)
0812 14:23:00.516441 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:00.516451 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:00.516483 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:00.516720 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:00.516721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:00.516722 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:00.516808 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.516809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.517563 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.517564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.518545 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:00.518548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.519448 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.519451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:00.519553 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.519556 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:23:00.519615 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:23:00.519651 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":55}
I20260812 14:23:00.520149 14289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01133b830f4d482a93150990395c6e66" candidate_uuid: "dec94ea6b5974186b3f220955b548336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62540953877147abab1c3c4206ff0bc7" is_pre_election: true
I20260812 14:23:00.520182 13976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01133b830f4d482a93150990395c6e66" candidate_uuid: "dec94ea6b5974186b3f220955b548336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" is_pre_election: true
I20260812 14:23:00.520313 14289 raft_consensus.cc:2454] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec94ea6b5974186b3f220955b548336 in term 0.
I20260812 14:23:00.520311 13976 raft_consensus.cc:2454] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec94ea6b5974186b3f220955b548336 in term 0.
I20260812 14:23:00.520538 14058 leader_election.cc:302] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bbd51a2cc8c4fe6af24d4ffda7f4054, dec94ea6b5974186b3f220955b548336; no voters:
I20260812 14:23:00.520704 14388 raft_consensus.cc:2781] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:23:00.520772 14388 raft_consensus.cc:482] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:23:00.520797 14388 raft_consensus.cc:3037] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:00.521580 14388 raft_consensus.cc:504] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } }
I20260812 14:23:00.521744 14388 leader_election.cc:288] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [CANDIDATE]: Term 1 election: Requested vote from peers 8bbd51a2cc8c4fe6af24d4ffda7f4054 (127.5.148.1:43061), 62540953877147abab1c3c4206ff0bc7 (127.5.148.3:39507)
I20260812 14:23:00.521940 13976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01133b830f4d482a93150990395c6e66" candidate_uuid: "dec94ea6b5974186b3f220955b548336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054"
I20260812 14:23:00.522043 13976 raft_consensus.cc:3037] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:00.522854 13976 raft_consensus.cc:2454] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec94ea6b5974186b3f220955b548336 in term 1.
I20260812 14:23:00.522994 14289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01133b830f4d482a93150990395c6e66" candidate_uuid: "dec94ea6b5974186b3f220955b548336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62540953877147abab1c3c4206ff0bc7"
I20260812 14:23:00.523036 14058 leader_election.cc:302] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bbd51a2cc8c4fe6af24d4ffda7f4054, dec94ea6b5974186b3f220955b548336; no voters:
I20260812 14:23:00.523108 14289 raft_consensus.cc:3037] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:00.523162 14388 raft_consensus.cc:2781] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:23:00.523329 14388 raft_consensus.cc:686] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [term 1 LEADER]: Becoming Leader. State: Replica: dec94ea6b5974186b3f220955b548336, State: Running, Role: LEADER
I20260812 14:23:00.523537 14388 consensus_queue.cc:237] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } }
I20260812 14:23:00.524096 14289 raft_consensus.cc:2454] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec94ea6b5974186b3f220955b548336 in term 1.
I20260812 14:23:00.524866 12884 catalog_manager.cc:5495] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 reported cstate change: term changed from 0 to 1, leader changed from <none> to dec94ea6b5974186b3f220955b548336 (127.5.148.2). New cstate: current_term: 1 leader_uuid: "dec94ea6b5974186b3f220955b548336" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } health_report { overall_health: HEALTHY } } }
W20260812 14:23:00.535380 14346 tablet.cc:2369] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:23:00.545392 14372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.542644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.5.148.2:38119)
0812 14:23:00.542718 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.542720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.542732 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:23:00.542821 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.542822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.542846 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.542939 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:23:00.543118 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.543672 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.544569 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.544763 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.544840 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.545033 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:23:00.545140 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:23:00.545230 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.545271 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.545295 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 14:23:00.547439 13976 raft_consensus.cc:1261] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 1 FOLLOWER]: Refusing update from remote peer dec94ea6b5974186b3f220955b548336: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:00.547472 14289 raft_consensus.cc:1261] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7 [term 1 FOLLOWER]: Refusing update from remote peer dec94ea6b5974186b3f220955b548336: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:00.547865 14389 consensus_queue.cc:1035] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:00.548048 14388 consensus_queue.cc:1035] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:00.551071 14394 mvcc.cc:205] Tried to move back new op lower bound from 7317686601919098880 to 7317686601825050624. Current Snapshot: MvccSnapshot[applied={T|T < 7317686601919098880}]
I20260812 14:23:00.552250 14397 mvcc.cc:205] Tried to move back new op lower bound from 7317686601919098880 to 7317686601825050624. Current Snapshot: MvccSnapshot[applied={T|T < 7317686601919098880}]
I20260812 14:23:00.552384 14395 mvcc.cc:205] Tried to move back new op lower bound from 7317686601919098880 to 7317686601825050624. Current Snapshot: MvccSnapshot[applied={T|T < 7317686601919098880 or (T in {7317686601919098880})}]
I20260812 14:23:00.554738 12884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:23:00.554855 12884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:23:00.556219 12884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8bac19a5015e4f9085a7a1c559f4a309: Sending DeleteTablet for 3 replicas of tablet 01133b830f4d482a93150990395c6e66
I20260812 14:23:00.556581 14106 tablet_service.cc:1514] Processing DeleteTablet for tablet 01133b830f4d482a93150990395c6e66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:00 UTC) from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:40590
I20260812 14:23:00.556834 13954 tablet_service.cc:1514] Processing DeleteTablet for tablet 01133b830f4d482a93150990395c6e66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:00 UTC) from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:55492
I20260812 14:23:00.557296 14268 tablet_service.cc:1514] Processing DeleteTablet for tablet 01133b830f4d482a93150990395c6e66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:00 UTC) from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:41720
I20260812 14:23:00.557454 14404 tablet_replica.cc:330] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054: stopping tablet replica
I20260812 14:23:00.557686 14404 raft_consensus.cc:2229] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:23:00.557909 14404 raft_consensus.cc:2258] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:00.557945 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 12843
I20260812 14:23:00.561156 14403 tablet_replica.cc:330] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336: stopping tablet replica
I20260812 14:23:00.561283 14403 raft_consensus.cc:2229] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:23:00.561424 14403 raft_consensus.cc:2258] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:00.561558 14404 ts_tablet_manager.cc:1905] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:00.561693 14406 tablet_replica.cc:330] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7: stopping tablet replica
I20260812 14:23:00.561805 14406 raft_consensus.cc:2229] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:23:00.561930 14406 raft_consensus.cc:2258] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:00.562274 14403 ts_tablet_manager.cc:1905] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:00.563572 14404 ts_tablet_manager.cc:1918] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:23:00.563699 14404 log.cc:1196] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-0/wal/wals/01133b830f4d482a93150990395c6e66
I20260812 14:23:00.564009 14404 ts_tablet_manager.cc:1939] T 01133b830f4d482a93150990395c6e66 P 8bbd51a2cc8c4fe6af24d4ffda7f4054: Deleting consensus metadata
I20260812 14:23:00.564486 14403 ts_tablet_manager.cc:1918] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:23:00.564575 14403 log.cc:1196] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/01133b830f4d482a93150990395c6e66
I20260812 14:23:00.564877 14403 ts_tablet_manager.cc:1939] T 01133b830f4d482a93150990395c6e66 P dec94ea6b5974186b3f220955b548336: Deleting consensus metadata
I20260812 14:23:00.565675 14406 ts_tablet_manager.cc:1905] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:00.567653 14406 ts_tablet_manager.cc:1918] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:23:00.567752 14406 log.cc:1196] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/01133b830f4d482a93150990395c6e66
I20260812 14:23:00.568066 14406 ts_tablet_manager.cc:1939] T 01133b830f4d482a93150990395c6e66 P 62540953877147abab1c3c4206ff0bc7: Deleting consensus metadata
W20260812 14:23:00.568241 13901 connection.cc:535] server connection from 127.0.0.1:55492 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 14:23:00.568248 14055 connection.cc:535] server connection from 127.0.0.1:40590 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 14:23:00.578786 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.570209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53846 (local address 127.5.148.61:35517)
0812 14:23:00.570259 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.570261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.570329 (+ 68us) server_negotiation.cc:407] Connection header received
0812 14:23:00.570391 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.570392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.570403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.570467 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:00.571399 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.572222 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.573328 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.573626 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.573672 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.575317 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:00.575319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:00.575320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:00.575327 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:00.577314 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.577899 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.577901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.577901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.577976 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.578181 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.578182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.578183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.578307 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:00.578377 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.578601 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.578683 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:23:00.578851 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.570134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.5.148.60:38841)
0812 14:23:00.570181 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.570183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.570308 (+ 125us) server_negotiation.cc:407] Connection header received
0812 14:23:00.570407 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.570408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.570420 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.570450 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:00.571767 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.572397 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.573469 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.573757 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.573793 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.575382 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:00.575383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:00.575384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:00.575391 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:00.577223 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.577842 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.577843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.577843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.577905 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.578126 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.578127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.578127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.578270 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:00.578365 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.578652 (+ 287us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.578735 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:23:00.581560 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:00.557154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.5.148.60:38841)
0812 14:23:00.557202 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:00.557205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:00.572299 (+ 15094us) server_negotiation.cc:407] Connection header received
0812 14:23:00.572392 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:00.572393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:00.572404 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:00.572436 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:00.573200 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.573904 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.575439 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:00.575715 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:00.575748 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:00.577903 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:00.577904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:00.577904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:00.577911 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:00.580491 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.580794 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.580795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.580795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.580848 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:00.581059 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:00.581060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:00.581060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:00.581166 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:00.581262 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:00.581375 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 14:23:00.581457 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:23:01.552990 14414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:01.552648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.1:49421)
0812 14:23:01.552833 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:01.552926 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":93,"threads_started":1}
W20260812 14:23:01.553254 14028 heartbeater.cc:644] Failed to heartbeat to 127.5.148.62:33907 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
I20260812 14:23:01.553301 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:01.553071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.2:57031)
0812 14:23:01.553175 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:01.553254 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":60,"threads_started":1}
W20260812 14:23:01.553508 14189 heartbeater.cc:644] Failed to heartbeat to 127.5.148.62:33907 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
I20260812 14:23:01.554454 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:01.554223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.3:57709)
0812 14:23:01.554331 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:01.554405 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
W20260812 14:23:01.554672 14344 heartbeater.cc:644] Failed to heartbeat to 127.5.148.62:33907 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
I20260812 14:23:02.056857 14418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8bac19a5015e4f9085a7a1c559f4a309)
I20260812 14:23:02.057017 14418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } }
I20260812 14:23:02.057356 14418 leader_election.cc:288] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8bac19a5015e4f9085a7a1c559f4a309 (127.5.148.62:33907), a56acc7854aa44b0bcf78df32f9bbbe2 (127.5.148.60:38841)
I20260812 14:23:02.057770 14419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.057493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.0.0.1:55570)
0812 14:23:02.057659 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.057726 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
W20260812 14:23:02.057956 13587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
I20260812 14:23:02.058004 14422 raft_consensus.cc:482] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8bac19a5015e4f9085a7a1c559f4a309)
I20260812 14:23:02.058140 14422 raft_consensus.cc:504] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } }
I20260812 14:23:02.058398 14419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.058304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.0.0.1:55576)
0812 14:23:02.058346 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.058368 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:23:02.058465 14422 leader_election.cc:288] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8bac19a5015e4f9085a7a1c559f4a309 (127.5.148.62:33907), 36dc93bfb0624fe8af2565d5bd070107 (127.5.148.61:35517)
W20260812 14:23:02.058681 13587 leader_election.cc:334] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8bac19a5015e4f9085a7a1c559f4a309 (127.5.148.62:33907): Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
I20260812 14:23:02.058739 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.058551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.0.0.1:55588)
0812 14:23:02.058655 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.058704 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":64,"threads_started":1}
W20260812 14:23:02.058835 13811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
I20260812 14:23:02.059271 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.059161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.0.0.1:55598)
0812 14:23:02.059199 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.059230 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 14:23:02.059358 13811 leader_election.cc:334] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8bac19a5015e4f9085a7a1c559f4a309 (127.5.148.62:33907): Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
I20260812 14:23:02.061337 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.057742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.5.148.60:38841)
0812 14:23:02.057932 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:02.057935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:02.057981 (+ 46us) server_negotiation.cc:407] Connection header received
0812 14:23:02.058015 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:02.058015 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:02.058030 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:02.058098 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:02.058326 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.059170 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.060578 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.060968 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.061039 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:02.061051 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:02.061216 (+ 165us) server_negotiation.cc:299] Negotiation successful
0812 14:23:02.061264 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":60,"threads_started":1}
I20260812 14:23:02.061409 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.057555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38841 (local address 127.0.0.1:49402)
0812 14:23:02.057748 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.057763 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:02.057925 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:02.058164 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:02.058165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:02.058168 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:02.058229 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:02.058231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.059244 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.059245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:02.060482 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:02.060486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.061063 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.061064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:02.061134 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:02.061139 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:23:02.061192 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 14:23:02.061332 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":94,"thread_start_us":47,"threads_started":1}
I20260812 14:23:02.061709 13835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36dc93bfb0624fe8af2565d5bd070107" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" is_pre_election: true
I20260812 14:23:02.061857 13835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36dc93bfb0624fe8af2565d5bd070107 in term 1.
I20260812 14:23:02.062088 13584 leader_election.cc:302] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36dc93bfb0624fe8af2565d5bd070107, a56acc7854aa44b0bcf78df32f9bbbe2; no voters: 8bac19a5015e4f9085a7a1c559f4a309
I20260812 14:23:02.062187 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.058595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35517 (local address 127.0.0.1:53848)
0812 14:23:02.058926 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.058938 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:02.059004 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:02.059180 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:02.059181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:02.059184 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:02.059236 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:02.059238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.060001 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.060002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:02.060992 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:02.060995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.061885 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.061887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:02.061971 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:02.061976 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:23:02.062058 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 14:23:02.062106 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":45,"threads_started":1}
I20260812 14:23:02.062213 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.058778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53848 (local address 127.5.148.61:35517)
0812 14:23:02.058944 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:02.058946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:02.058986 (+ 40us) server_negotiation.cc:407] Connection header received
0812 14:23:02.059053 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:02.059054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:02.059064 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:02.059130 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:02.059307 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.059916 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.061117 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.061769 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.061852 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:02.061866 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:02.062065 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 14:23:02.062110 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260812 14:23:02.062260 14418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 14:23:02.062485 14418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8bac19a5015e4f9085a7a1c559f4a309)
I20260812 14:23:02.062525 14418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:23:02.062634 13614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "36dc93bfb0624fe8af2565d5bd070107" is_pre_election: true
I20260812 14:23:02.063380 14418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } }
I20260812 14:23:02.063530 13614 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a56acc7854aa44b0bcf78df32f9bbbe2 in current term 2: Already voted for candidate 36dc93bfb0624fe8af2565d5bd070107 in this term.
I20260812 14:23:02.063560 14418 leader_election.cc:288] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [CANDIDATE]: Term 2 election: Requested vote from peers 8bac19a5015e4f9085a7a1c559f4a309 (127.5.148.62:33907), a56acc7854aa44b0bcf78df32f9bbbe2 (127.5.148.60:38841)
I20260812 14:23:02.063755 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.063657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.0.0.1:55600)
0812 14:23:02.063694 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.063727 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260812 14:23:02.063755 13835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36dc93bfb0624fe8af2565d5bd070107" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2"
I20260812 14:23:02.063792 13809 leader_election.cc:302] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a56acc7854aa44b0bcf78df32f9bbbe2; no voters: 36dc93bfb0624fe8af2565d5bd070107, 8bac19a5015e4f9085a7a1c559f4a309
I20260812 14:23:02.063835 13835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:23:02.064085 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.063985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.0.0.1:55616)
0812 14:23:02.064035 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.064060 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 14:23:02.064189 13587 leader_election.cc:334] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8bac19a5015e4f9085a7a1c559f4a309 (127.5.148.62:33907): Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
I20260812 14:23:02.064633 13835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36dc93bfb0624fe8af2565d5bd070107 in term 2.
I20260812 14:23:02.064760 14422 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:23:02.064852 13584 leader_election.cc:302] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36dc93bfb0624fe8af2565d5bd070107, a56acc7854aa44b0bcf78df32f9bbbe2; no voters: 8bac19a5015e4f9085a7a1c559f4a309
I20260812 14:23:02.064986 14418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 14:23:02.065146 14418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [term 2 LEADER]: Becoming Leader. State: Replica: 36dc93bfb0624fe8af2565d5bd070107, State: Running, Role: LEADER
I20260812 14:23:02.065214 14418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } }
I20260812 14:23:02.065584 14429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36dc93bfb0624fe8af2565d5bd070107. Latest consensus state: current_term: 2 leader_uuid: "36dc93bfb0624fe8af2565d5bd070107" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:23:02.065676 14429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: This master's current role is: LEADER
I20260812 14:23:02.065863 14431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:23:02.066365 14431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:23:02.066438 14431 catalog_manager.cc:1259] Loaded cluster ID: a52194445a7e4e6ca42c123286c3fa53
I20260812 14:23:02.066458 14431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:23:02.066594 14431 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:23:02.066699 14431 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: Loaded TSK: 0
I20260812 14:23:02.066843 14431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:23:02.114530 13601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:23:02.116350 13601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:23:02.117683 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.117515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.0.0.1:55630)
0812 14:23:02.117605 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.117648 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 14:23:02.117693 13835 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [term 2 FOLLOWER]: Refusing update from remote peer 36dc93bfb0624fe8af2565d5bd070107: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260812 14:23:02.117951 14418 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260812 14:23:02.118103 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.117962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.0.0.1:55634)
0812 14:23:02.118011 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.118042 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 14:23:02.118171 13587 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 -> Peer 8bac19a5015e4f9085a7a1c559f4a309 (127.5.148.62:33907): Couldn't send request to peer 8bac19a5015e4f9085a7a1c559f4a309. Status: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 14:23:02.118719 14422 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36dc93bfb0624fe8af2565d5bd070107. Latest consensus state: current_term: 2 leader_uuid: "36dc93bfb0624fe8af2565d5bd070107" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:23:02.118863 14422 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:23:02.119288 14433 mvcc.cc:205] Tried to move back new op lower bound from 7317686608351510528 to 7317686608139771904. Current Snapshot: MvccSnapshot[applied={T|T < 7317686608351510528}]
I20260812 14:23:02.119380 14429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "36dc93bfb0624fe8af2565d5bd070107" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:23:02.119482 14429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107 [sys.catalog]: This master's current role is: LEADER
I20260812 14:23:02.119692 14422 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "36dc93bfb0624fe8af2565d5bd070107" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8bac19a5015e4f9085a7a1c559f4a309" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33907 } } peers { permanent_uuid: "36dc93bfb0624fe8af2565d5bd070107" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35517 } attrs { promote: false } } peers { permanent_uuid: "a56acc7854aa44b0bcf78df32f9bbbe2" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38841 } attrs { promote: false } } }
I20260812 14:23:02.119812 14422 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56acc7854aa44b0bcf78df32f9bbbe2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:23:02.124837 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.121521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41722 (local address 127.5.148.3:39507)
0812 14:23:02.121633 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:02.121636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:02.121644 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:23:02.121674 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:02.121675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:02.121687 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:02.121787 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:02.122016 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.122605 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.123863 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.124450 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.124542 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:02.124557 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:02.124721 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 14:23:02.124765 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":65,"threads_started":1}
I20260812 14:23:02.124859 14419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.121393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:39507 (local address 127.0.0.1:41722)
0812 14:23:02.121512 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.121522 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:02.121576 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:02.121797 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:02.121799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:02.121802 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:02.121888 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:02.121890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.122699 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.122700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:02.123762 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:02.123764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.124541 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.124543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:02.124620 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:02.124624 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:23:02.124710 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 14:23:02.124757 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260812 14:23:02.125262 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.121414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:43061 (local address 127.0.0.1:55506)
0812 14:23:02.121702 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.121712 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:02.121750 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:02.122130 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:02.122131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:02.122133 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:02.122181 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:02.122183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.123068 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.123069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:02.124193 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:02.124196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.124989 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.124990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:02.125058 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:02.125063 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:23:02.125130 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:23:02.125177 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":23,"thread_start_us":76,"threads_started":1}
I20260812 14:23:02.125290 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.121599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55506 (local address 127.5.148.1:43061)
0812 14:23:02.121724 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:02.121726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:02.121750 (+ 24us) server_negotiation.cc:407] Connection header received
0812 14:23:02.122000 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:02.122001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:02.122016 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:02.122081 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:02.122268 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.122986 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.124278 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.124863 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.124933 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:02.124950 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:02.125166 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 14:23:02.125221 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260812 14:23:02.125265 14268 tablet_service.cc:1467] Processing CreateTablet for tablet 1da1f91820714a55b9eb865082bdce9a (DEFAULT_TABLE table=test-table [id=6e1ef0d2c117442285793384106afb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:02.125428 14268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1da1f91820714a55b9eb865082bdce9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:02.125592 13954 tablet_service.cc:1467] Processing CreateTablet for tablet 1da1f91820714a55b9eb865082bdce9a (DEFAULT_TABLE table=test-table [id=6e1ef0d2c117442285793384106afb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:02.125712 13954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1da1f91820714a55b9eb865082bdce9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:02.125811 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.121316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:38119 (local address 127.0.0.1:40606)
0812 14:23:02.121397 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.121409 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:02.121462 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:02.122068 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:02.122069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:02.122071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:02.122109 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:02.122111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.123076 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.123077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:02.124040 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:02.124043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.124857 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.124858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:02.125628 (+ 770us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:02.125633 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:23:02.125697 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 14:23:02.125742 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":54}
I20260812 14:23:02.126362 14439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.121411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40606 (local address 127.5.148.2:38119)
0812 14:23:02.121934 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:02.121937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:02.121945 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:23:02.121975 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:02.121976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:02.121990 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:02.122066 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:02.122175 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.122959 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.124156 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.124741 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.126074 (+ 1333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:02.126094 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:02.126114 (+ 20us) server_negotiation.cc:299] Negotiation successful
0812 14:23:02.126280 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":70,"threads_started":1}
I20260812 14:23:02.126583 14106 tablet_service.cc:1467] Processing CreateTablet for tablet 1da1f91820714a55b9eb865082bdce9a (DEFAULT_TABLE table=test-table [id=6e1ef0d2c117442285793384106afb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:02.126742 14106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1da1f91820714a55b9eb865082bdce9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:02.127036 14443 tablet_bootstrap.cc:492] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7: Bootstrap starting.
I20260812 14:23:02.127620 14443 tablet_bootstrap.cc:654] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:02.128263 14444 tablet_bootstrap.cc:492] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054: Bootstrap starting.
I20260812 14:23:02.128348 14443 tablet_bootstrap.cc:492] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7: No bootstrap required, opened a new log
I20260812 14:23:02.128413 14443 ts_tablet_manager.cc:1397] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:02.128628 14444 tablet_bootstrap.cc:654] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:02.128593 14443 raft_consensus.cc:348] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } }
I20260812 14:23:02.128682 14443 raft_consensus.cc:374] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:02.128700 14443 raft_consensus.cc:729] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62540953877147abab1c3c4206ff0bc7, State: Initialized, Role: FOLLOWER
I20260812 14:23:02.128795 14443 consensus_queue.cc:260] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } }
I20260812 14:23:02.128985 14443 ts_tablet_manager.cc:1428] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 14:23:02.129395 14447 tablet_bootstrap.cc:492] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336: Bootstrap starting.
I20260812 14:23:02.129415 14444 tablet_bootstrap.cc:492] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054: No bootstrap required, opened a new log
I20260812 14:23:02.129467 14444 ts_tablet_manager.cc:1397] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:02.129626 14444 raft_consensus.cc:348] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } }
I20260812 14:23:02.129703 14444 raft_consensus.cc:374] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:02.129732 14444 raft_consensus.cc:729] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bbd51a2cc8c4fe6af24d4ffda7f4054, State: Initialized, Role: FOLLOWER
I20260812 14:23:02.129781 14444 consensus_queue.cc:260] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } }
I20260812 14:23:02.129961 14447 tablet_bootstrap.cc:654] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:02.129958 14444 ts_tablet_manager.cc:1428] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:23:02.130095 14345 heartbeater.cc:503] Master 127.5.148.61:35517 was elected leader, sending a full tablet report...
I20260812 14:23:02.130373 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.129531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.3:48189)
0812 14:23:02.130277 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.130328 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":85,"threads_started":1}
I20260812 14:23:02.130609 14447 tablet_bootstrap.cc:492] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336: No bootstrap required, opened a new log
I20260812 14:23:02.130673 14447 ts_tablet_manager.cc:1397] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:02.130707 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.130462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.1:60533)
0812 14:23:02.130626 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.130670 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260812 14:23:02.130841 14029 heartbeater.cc:503] Master 127.5.148.61:35517 was elected leader, sending a full tablet report...
I20260812 14:23:02.130805 14447 raft_consensus.cc:348] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } }
I20260812 14:23:02.130882 14447 raft_consensus.cc:374] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:02.130965 14447 raft_consensus.cc:729] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec94ea6b5974186b3f220955b548336, State: Initialized, Role: FOLLOWER
I20260812 14:23:02.131024 14447 consensus_queue.cc:260] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } }
I20260812 14:23:02.131188 14447 ts_tablet_manager.cc:1428] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:23:02.131562 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.131119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.1:50277)
0812 14:23:02.131157 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.131190 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260812 14:23:02.131714 14190 heartbeater.cc:503] Master 127.5.148.61:35517 was elected leader, sending a full tablet report...
I20260812 14:23:02.132035 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.131946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.3:53791)
0812 14:23:02.131982 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.132006 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260812 14:23:02.132205 14458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.131393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.2:35603)
0812 14:23:02.131894 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.131936 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":83,"threads_started":1}
I20260812 14:23:02.132688 14458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.132572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.2:34895)
0812 14:23:02.132632 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.132660 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 14:23:02.169466 14452 raft_consensus.cc:482] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:23:02.169581 14452 raft_consensus.cc:504] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } }
I20260812 14:23:02.169922 14452 leader_election.cc:288] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec94ea6b5974186b3f220955b548336 (127.5.148.2:38119), 62540953877147abab1c3c4206ff0bc7 (127.5.148.3:39507)
I20260812 14:23:02.173134 14439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.170227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:53609 (local address 127.5.148.2:38119)
0812 14:23:02.170280 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:02.170283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:02.170564 (+ 281us) server_negotiation.cc:407] Connection header received
0812 14:23:02.170601 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:02.170602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:02.170617 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:02.170682 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:02.170853 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.171310 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.172390 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.172746 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.172803 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:02.172813 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:02.173004 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 14:23:02.173054 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:23:02.173184 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.170231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:49047 (local address 127.5.148.3:39507)
0812 14:23:02.170292 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:02.170294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:02.170354 (+ 60us) server_negotiation.cc:407] Connection header received
0812 14:23:02.170446 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:02.170447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:02.170461 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:02.170542 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:02.170748 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.171282 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.172331 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.172749 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.172858 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:02.172870 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:02.173065 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 14:23:02.173112 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:23:02.173170 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.170140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:39507 (local address 127.5.148.1:49047)
0812 14:23:02.170271 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.170283 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:02.170357 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:02.170577 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:02.170579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:02.170582 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:02.170671 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:02.170674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.171369 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.171371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:02.172243 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:02.172246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.172908 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.172909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:02.172980 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:02.172984 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:23:02.173043 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:23:02.173080 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":2}
I20260812 14:23:02.173164 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.170148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:38119 (local address 127.5.148.1:53609)
0812 14:23:02.170474 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.170486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:02.170551 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:02.170691 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:02.170691 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:02.170693 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:02.170779 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:02.170781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.171419 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.171420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:02.172267 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:02.172270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.172831 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.172833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:02.172921 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:02.172925 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:23:02.172981 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 14:23:02.173015 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":64,"thread_start_us":184,"threads_started":1}
I20260812 14:23:02.173418 14289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da1f91820714a55b9eb865082bdce9a" candidate_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62540953877147abab1c3c4206ff0bc7" is_pre_election: true
I20260812 14:23:02.173539 14289 raft_consensus.cc:2454] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bbd51a2cc8c4fe6af24d4ffda7f4054 in term 0.
I20260812 14:23:02.173655 14130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da1f91820714a55b9eb865082bdce9a" candidate_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec94ea6b5974186b3f220955b548336" is_pre_election: true
I20260812 14:23:02.173738 13902 leader_election.cc:302] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62540953877147abab1c3c4206ff0bc7, 8bbd51a2cc8c4fe6af24d4ffda7f4054; no voters:
I20260812 14:23:02.173830 14130 raft_consensus.cc:2454] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bbd51a2cc8c4fe6af24d4ffda7f4054 in term 0.
I20260812 14:23:02.173908 14452 raft_consensus.cc:2781] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:23:02.173956 14452 raft_consensus.cc:482] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:23:02.173978 14452 raft_consensus.cc:3037] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:02.174748 14452 raft_consensus.cc:504] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } }
I20260812 14:23:02.174903 14452 leader_election.cc:288] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [CANDIDATE]: Term 1 election: Requested vote from peers dec94ea6b5974186b3f220955b548336 (127.5.148.2:38119), 62540953877147abab1c3c4206ff0bc7 (127.5.148.3:39507)
I20260812 14:23:02.175046 14130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da1f91820714a55b9eb865082bdce9a" candidate_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec94ea6b5974186b3f220955b548336"
I20260812 14:23:02.175137 14130 raft_consensus.cc:3037] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:02.175122 14289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da1f91820714a55b9eb865082bdce9a" candidate_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62540953877147abab1c3c4206ff0bc7"
I20260812 14:23:02.175235 14289 raft_consensus.cc:3037] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:02.175818 14289 raft_consensus.cc:2454] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bbd51a2cc8c4fe6af24d4ffda7f4054 in term 1.
I20260812 14:23:02.175997 13902 leader_election.cc:302] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62540953877147abab1c3c4206ff0bc7, 8bbd51a2cc8c4fe6af24d4ffda7f4054; no voters:
I20260812 14:23:02.176016 14130 raft_consensus.cc:2454] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bbd51a2cc8c4fe6af24d4ffda7f4054 in term 1.
I20260812 14:23:02.176108 14452 raft_consensus.cc:2781] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:23:02.176307 14452 raft_consensus.cc:686] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 1 LEADER]: Becoming Leader. State: Replica: 8bbd51a2cc8c4fe6af24d4ffda7f4054, State: Running, Role: LEADER
I20260812 14:23:02.176412 14452 consensus_queue.cc:237] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } } peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } }
I20260812 14:23:02.176800 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.176677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.1:35123)
0812 14:23:02.176731 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.176767 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 14:23:02.177254 13600 catalog_manager.cc:5495] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bbd51a2cc8c4fe6af24d4ffda7f4054 (127.5.148.1). New cstate: current_term: 1 leader_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bbd51a2cc8c4fe6af24d4ffda7f4054" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43061 } health_report { overall_health: HEALTHY } } }
I20260812 14:23:02.198951 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.196422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55510 (local address 127.5.148.1:43061)
0812 14:23:02.196495 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:02.196497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:02.196506 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:23:02.196563 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:02.196564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:02.196588 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:02.196667 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:23:02.196816 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.197292 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.198069 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:02.198270 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:02.198344 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:02.198590 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:23:02.198716 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:23:02.198811 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:02.198835 (+ 24us) server_negotiation.cc:299] Negotiation successful
0812 14:23:02.198859 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 14:23:02.200268 14289 raft_consensus.cc:1261] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7 [term 1 FOLLOWER]: Refusing update from remote peer 8bbd51a2cc8c4fe6af24d4ffda7f4054: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:02.200311 14130 raft_consensus.cc:1261] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336 [term 1 FOLLOWER]: Refusing update from remote peer 8bbd51a2cc8c4fe6af24d4ffda7f4054: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:02.200613 14461 consensus_queue.cc:1035] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62540953877147abab1c3c4206ff0bc7" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:02.200763 14452 consensus_queue.cc:1035] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec94ea6b5974186b3f220955b548336" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:02.201483 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.201369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.3:59209)
0812 14:23:02.201414 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.201448 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:23:02.201952 14458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.201849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.2:50655)
0812 14:23:02.201893 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.201920 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 14:23:02.202994 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.202758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.1:53111)
0812 14:23:02.202927 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.202958 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260812 14:23:02.203038 14458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.202943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.2:35699)
0812 14:23:02.202980 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.203009 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 14:23:02.204339 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.204230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.3:48235)
0812 14:23:02.204270 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.204296 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260812 14:23:02.204622 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.204466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33907 (local address 127.5.148.1:42339)
0812 14:23:02.204563 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.204591 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 14:23:02.204738 13599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:23:02.204859 13599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:23:02.206218 13599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36dc93bfb0624fe8af2565d5bd070107: Sending DeleteTablet for 3 replicas of tablet 1da1f91820714a55b9eb865082bdce9a
I20260812 14:23:02.206527 14268 tablet_service.cc:1514] Processing DeleteTablet for tablet 1da1f91820714a55b9eb865082bdce9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:02 UTC) from {username='oryx', principal='oryx/127.5.148.61@KRBTEST.COM'} at 127.0.0.1:41722
I20260812 14:23:02.206607 13954 tablet_service.cc:1514] Processing DeleteTablet for tablet 1da1f91820714a55b9eb865082bdce9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:02 UTC) from {username='oryx', principal='oryx/127.5.148.61@KRBTEST.COM'} at 127.0.0.1:55506
I20260812 14:23:02.206782 14106 tablet_service.cc:1514] Processing DeleteTablet for tablet 1da1f91820714a55b9eb865082bdce9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:02 UTC) from {username='oryx', principal='oryx/127.5.148.61@KRBTEST.COM'} at 127.0.0.1:40606
I20260812 14:23:02.206847 14474 tablet_replica.cc:330] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054: stopping tablet replica
I20260812 14:23:02.206902 14474 raft_consensus.cc:2229] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:23:02.206984 14474 raft_consensus.cc:2258] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:02.207099 14475 tablet_replica.cc:330] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336: stopping tablet replica
I20260812 14:23:02.207151 14475 raft_consensus.cc:2229] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:23:02.207199 14475 raft_consensus.cc:2258] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:02.207301 14474 ts_tablet_manager.cc:1905] T 1da1f91820714a55b9eb865082bdce9a P 8bbd51a2cc8c4fe6af24d4ffda7f4054: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:02.207365 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 13784
I20260812 14:23:02.209437 14473 tablet_replica.cc:330] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7: stopping tablet replica
I20260812 14:23:02.209491 14473 raft_consensus.cc:2229] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:23:02.209532 14473 raft_consensus.cc:2258] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:02.210165 14473 ts_tablet_manager.cc:1905] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:02.213080 14473 ts_tablet_manager.cc:1918] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:23:02.213145 14473 log.cc:1196] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/1da1f91820714a55b9eb865082bdce9a
I20260812 14:23:02.213364 14473 ts_tablet_manager.cc:1939] T 1da1f91820714a55b9eb865082bdce9a P 62540953877147abab1c3c4206ff0bc7: Deleting consensus metadata
I20260812 14:23:02.214396 14475 ts_tablet_manager.cc:1905] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:02.214393 13585 catalog_manager.cc:4841] TS 62540953877147abab1c3c4206ff0bc7 (127.5.148.3:39507): tablet 1da1f91820714a55b9eb865082bdce9a (table test-table [id=6e1ef0d2c117442285793384106afb4f]) successfully deleted
I20260812 14:23:02.216943 14475 ts_tablet_manager.cc:1918] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:23:02.217011 14475 log.cc:1196] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/1da1f91820714a55b9eb865082bdce9a
I20260812 14:23:02.217303 14475 ts_tablet_manager.cc:1939] T 1da1f91820714a55b9eb865082bdce9a P dec94ea6b5974186b3f220955b548336: Deleting consensus metadata
I20260812 14:23:02.217685 13587 catalog_manager.cc:4841] TS dec94ea6b5974186b3f220955b548336 (127.5.148.2:38119): tablet 1da1f91820714a55b9eb865082bdce9a (table test-table [id=6e1ef0d2c117442285793384106afb4f]) successfully deleted
I20260812 14:23:02.222173 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:02.222032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:43061 (local address 127.0.0.1:55526)
0812 14:23:02.222097 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:02.222131 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:43061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 14:23:02.222183 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 14036
W20260812 14:23:02.224313 13587 catalog_manager.cc:4584] TS 8bbd51a2cc8c4fe6af24d4ffda7f4054 (127.5.148.1:43061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1da1f91820714a55b9eb865082bdce9a: Network error: Client connection negotiation failed: client connection to 127.5.148.1:43061: connect: Connection refused (error 111)
I20260812 14:23:02.233889 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 14196
I20260812 14:23:02.242342 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 13450
I20260812 14:23:02.251024 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 13664
2026-08-12T14:23:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5010 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:23:02 dist-test-slave-8w9v krb5kdc[14478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:23:02 dist-test-slave-8w9v krb5kdc[14478](info): set up 2 sockets
Aug 12 14:23:02 dist-test-slave-8w9v krb5kdc[14478](info): commencing operation
krb5kdc: starting...
W20260812 14:23:03.390209 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:23:03 dist-test-slave-8w9v krb5kdc[14478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:23:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:23:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.62.keytab.
Entry for principal oryx/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.62.keytab.
I20260812 14:23:03.526471 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:38971
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.62.keytab
--principal=oryx/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:38971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:23:03.611971 14494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:03.612277 14494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:03.612306 14494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:03.613507 14494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:23:03.613550 14494 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 14:23:03.613564 14494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:03.613577 14494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:03.613590 14494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:23:03.613647 14494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:23:03.615185 14494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:38971
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.62.keytab
--principal=oryx/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:38971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:03.615473 14494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:03.615715 14494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:03.621440 14503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:03.621435 14499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:03.621435 14501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:03.621469 14494 server_base.cc:1034] running on GCE node
I20260812 14:23:03.621872 14494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:03.622153 14494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:03.623306 14494 hybrid_clock.cc:648] HybridClock initialized: now 1786544583623290 us; error 31 us; skew 500 ppm
Aug 12 14:23:03 dist-test-slave-8w9v krb5kdc[14478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544583, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:03.626202 14494 init.cc:377] Logged in from keytab as oryx/127.5.148.62@KRBTEST.COM (short username oryx)
I20260812 14:23:03.626680 14494 webserver.cc:460] Webserver started at http://127.5.148.62:41487/ using document root <none> and password file <none>
I20260812 14:23:03.626940 14494 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:03.627012 14494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:03.627166 14494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:03.628376 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "0cd451a6a91140c2be387269249dbdcb"
format_stamp: "Formatted at 2026-08-12 14:23:03 on dist-test-slave-8w9v"
server_key: "b724fc4dca68ea0013ac20bd0c6a5b98"
server_key_iv: "8e0cb375e6f52046556ee51818c2f2c5"
server_key_version: "encryptionkey@0"
I20260812 14:23:03.628759 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "0cd451a6a91140c2be387269249dbdcb"
format_stamp: "Formatted at 2026-08-12 14:23:03 on dist-test-slave-8w9v"
server_key: "b724fc4dca68ea0013ac20bd0c6a5b98"
server_key_iv: "8e0cb375e6f52046556ee51818c2f2c5"
server_key_version: "encryptionkey@0"
I20260812 14:23:03.630298 14494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:23:03.631140 14513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:03.631364 14494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:03.631491 14494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "0cd451a6a91140c2be387269249dbdcb"
format_stamp: "Formatted at 2026-08-12 14:23:03 on dist-test-slave-8w9v"
server_key: "b724fc4dca68ea0013ac20bd0c6a5b98"
server_key_iv: "8e0cb375e6f52046556ee51818c2f2c5"
server_key_version: "encryptionkey@0"
I20260812 14:23:03.631577 14494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:03.651702 14494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:03.653996 14494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:03.654176 14494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:03.659535 14494 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:38971
I20260812 14:23:03.659626 14588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:38971 every 8 connection(s)
I20260812 14:23:03.659988 14494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:23:03.660760 14590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:03.662567 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 14494
I20260812 14:23:03.662672 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:23:03.662856 5712 external_mini_cluster.cc:1428] Setting key 9d0ed667e042c02a39860a97264071b2
I20260812 14:23:03.663239 14590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb: Bootstrap starting.
I20260812 14:23:03.664283 14590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:03.664660 14590 log.cc:824] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb: Log is configured to *not* fsync() on all Append() calls
I20260812 14:23:03.665441 14590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb: No bootstrap required, opened a new log
I20260812 14:23:03.667816 14590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cd451a6a91140c2be387269249dbdcb" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38971 } }
I20260812 14:23:03.668002 14590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:03.668042 14590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cd451a6a91140c2be387269249dbdcb, State: Initialized, Role: FOLLOWER
I20260812 14:23:03.668227 14590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cd451a6a91140c2be387269249dbdcb" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38971 } }
I20260812 14:23:03.668337 14590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:23:03.668375 14590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:23:03.668423 14590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:03.669094 14590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cd451a6a91140c2be387269249dbdcb" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38971 } }
I20260812 14:23:03.669189 14590 leader_election.cc:302] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cd451a6a91140c2be387269249dbdcb; no voters:
I20260812 14:23:03.669431 14590 leader_election.cc:288] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:23:03.669457 14596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:23:03.669574 14596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [term 1 LEADER]: Becoming Leader. State: Replica: 0cd451a6a91140c2be387269249dbdcb, State: Running, Role: LEADER
I20260812 14:23:03.669668 14596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cd451a6a91140c2be387269249dbdcb" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38971 } }
I20260812 14:23:03.669710 14590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:23:03.670006 14596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cd451a6a91140c2be387269249dbdcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cd451a6a91140c2be387269249dbdcb" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38971 } } }
I20260812 14:23:03.670077 14596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [sys.catalog]: This master's current role is: LEADER
I20260812 14:23:03.670197 14596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cd451a6a91140c2be387269249dbdcb. Latest consensus state: current_term: 1 leader_uuid: "0cd451a6a91140c2be387269249dbdcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cd451a6a91140c2be387269249dbdcb" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 38971 } } }
I20260812 14:23:03.670252 14596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb [sys.catalog]: This master's current role is: LEADER
Aug 12 14:23:03 dist-test-slave-8w9v krb5kdc[14478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
W20260812 14:23:03.671528 14615 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:23:03.671893 14615 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:23:03.671974 14601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:23:03.672175 14601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:23:03.673759 14601 catalog_manager.cc:1347] Generated new cluster ID: 286b724cad114616820a75e4fb21a15e
I20260812 14:23:03.673820 14601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:23:03.675158 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:03.663418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58454 (local address 127.5.148.62:38971)
0812 14:23:03.663590 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:03.663599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:03.663634 (+ 35us) server_negotiation.cc:407] Connection header received
0812 14:23:03.664303 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:03.664313 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:03.664383 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:03.664566 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:03.665226 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:03.666726 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:03.667445 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:03.668071 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:03.668247 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:03.671178 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:03.671182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:03.671182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:03.671193 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:03.673225 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:03.673633 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:03.673636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:03.673636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:03.673746 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:03.673983 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:03.673984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:03.673984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:03.674267 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:03.674393 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:03.674815 (+ 422us) server_negotiation.cc:299] Negotiation successful
0812 14:23:03.674950 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260812 14:23:03.685652 14601 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:23:03.686219 14601 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:23:03.690765 14601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb: Generated new TSK 0
I20260812 14:23:03.690965 14601 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.1.keytab.
Entry for principal oryx/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.1.keytab.
I20260812 14:23:03.735426 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:38971
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.1.keytab
--principal=oryx/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:23:03.849592 14620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:03.849867 14620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:03.849944 14620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:03.851198 14620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:03.851274 14620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:03.851307 14620 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:23:03.851351 14620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:23:03.853027 14620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.1.keytab
--principal=oryx/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.148.62:38971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:03.853298 14620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:03.853543 14620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:03.858945 14627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:03.858974 14629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:03.858943 14626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:03.859244 14620 server_base.cc:1034] running on GCE node
I20260812 14:23:03.859498 14620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:03.859746 14620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:03.860922 14620 hybrid_clock.cc:648] HybridClock initialized: now 1786544583860897 us; error 43 us; skew 500 ppm
Aug 12 14:23:03 dist-test-slave-8w9v krb5kdc[14478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544583, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:03.863682 14620 init.cc:377] Logged in from keytab as oryx/127.5.148.1@KRBTEST.COM (short username oryx)
I20260812 14:23:03.864287 14620 webserver.cc:460] Webserver started at http://127.5.148.1:35627/ using document root <none> and password file <none>
I20260812 14:23:03.864534 14620 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:03.864602 14620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:03.864741 14620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:03.865877 14620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "ce4ce81490bf41de8b8f6747be5e945a"
format_stamp: "Formatted at 2026-08-12 14:23:03 on dist-test-slave-8w9v"
server_key: "9dc4a093d8fb2d579208ef7f3175e468"
server_key_iv: "2302b9a3b494b7b7d4ffd568c20c171b"
server_key_version: "encryptionkey@0"
I20260812 14:23:03.866273 14620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "ce4ce81490bf41de8b8f6747be5e945a"
format_stamp: "Formatted at 2026-08-12 14:23:03 on dist-test-slave-8w9v"
server_key: "9dc4a093d8fb2d579208ef7f3175e468"
server_key_iv: "2302b9a3b494b7b7d4ffd568c20c171b"
server_key_version: "encryptionkey@0"
I20260812 14:23:03.867792 14620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:23:03.868999 14637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:03.869189 14620 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:03.869349 14620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "ce4ce81490bf41de8b8f6747be5e945a"
format_stamp: "Formatted at 2026-08-12 14:23:03 on dist-test-slave-8w9v"
server_key: "9dc4a093d8fb2d579208ef7f3175e468"
server_key_iv: "2302b9a3b494b7b7d4ffd568c20c171b"
server_key_version: "encryptionkey@0"
I20260812 14:23:03.869442 14620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:03.881937 14620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:03.884337 14620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:03.884521 14620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:03.885142 14620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:03.885205 14620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:03.885259 14620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:03.885288 14620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:23:03 dist-test-slave-8w9v krb5kdc[14478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544583, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
I20260812 14:23:03.893412 14620 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:41081
I20260812 14:23:03.893527 14770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:41081 every 8 connection(s)
I20260812 14:23:03.893890 14620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:23:03.896404 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:03.885683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:53273 (local address 127.5.148.62:38971)
0812 14:23:03.885747 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:03.885749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:03.886037 (+ 288us) server_negotiation.cc:407] Connection header received
0812 14:23:03.886644 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:03.886646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:03.886662 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:03.886727 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:03.887500 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:03.888376 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:03.889517 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:03.889778 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:03.889867 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:03.893201 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:03.893203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:03.893204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:03.893214 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:03.894991 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:03.895464 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:03.895465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:03.895465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:03.895527 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:03.895771 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:03.895771 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:03.895771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:03.895883 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:03.895941 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:03.896255 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 14:23:03.896339 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:23:03.896622 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:03.885665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38971 (local address 127.5.148.1:53273)
0812 14:23:03.885855 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:03.885894 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:03.886511 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:03.886814 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:03.886818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:03.887156 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:03.887386 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:03.887392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:03.888485 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:03.888486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:03.889407 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:03.889409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:03.889931 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:03.889937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:03.890116 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:03.890758 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:03.890768 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:03.893078 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:03.895074 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:03.895080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:03.895082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:03.895357 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:03.895602 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:03.895603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:03.895603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:03.895709 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:03.895929 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:03.895933 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:03.896067 (+ 134us) client_negotiation.cc:769] Sending connection context
0812 14:23:03.896225 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 14:23:03.896407 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
I20260812 14:23:03.897208 14771 heartbeater.cc:347] Connected to a master server at 127.5.148.62:38971
I20260812 14:23:03.897297 14771 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:03.897495 14771 heartbeater.cc:511] Master 127.5.148.62:38971 requested a full tablet report, sending...
I20260812 14:23:03.898144 14534 ts_manager.cc:194] Registered new tserver with Master: ce4ce81490bf41de8b8f6747be5e945a (127.5.148.1:41081)
I20260812 14:23:03.899013 14534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:53273
I20260812 14:23:03.903308 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 14620
I20260812 14:23:03.903427 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:23:03.903663 5712 external_mini_cluster.cc:1428] Setting key b7ee8ab9f2d1077db822c5551b5fce42
I20260812 14:23:03.904142 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:03.898442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:51025 (local address 127.5.148.62:38971)
0812 14:23:03.898483 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:03.898485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:03.898633 (+ 148us) server_negotiation.cc:407] Connection header received
0812 14:23:03.898664 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:03.898665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:03.898674 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:03.898743 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:03.899028 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:03.899659 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:03.900439 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:03.900620 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:03.900679 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:03.901626 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:03.901627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:03.901628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:03.901632 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:03.902813 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:03.903176 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:03.903177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:03.903178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:03.903246 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:03.903520 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:03.903521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:03.903522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:03.903663 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:03.903739 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:03.903946 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 14:23:03.904027 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:23:03.904158 14777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:03.898393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38971 (local address 127.5.148.1:51025)
0812 14:23:03.898554 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:03.898565 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:03.898627 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:03.898745 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:03.898746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:03.898899 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:03.898963 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:03.898965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:03.899760 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:03.899761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:03.900364 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:03.900366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:03.900689 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:03.900689 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:03.900740 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:03.901239 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:03.901244 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:03.901552 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:03.902920 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:03.902921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:03.902921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:03.903097 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:03.903343 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:03.903344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:03.903344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:03.903446 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:03.903750 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:03.903751 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:03.903818 (+ 67us) client_negotiation.cc:769] Sending connection context
0812 14:23:03.903914 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 14:23:03.904030 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for oryx/127.5.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.2.keytab.
Entry for principal oryx/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.2.keytab.
I20260812 14:23:03.931627 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:38971
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.2.keytab
--principal=oryx/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:23:04.019999 14780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:04.020375 14780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:04.020416 14780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:04.021818 14780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:04.021871 14780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:04.021894 14780 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:23:04.021944 14780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:23:04.023641 14780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.2.keytab
--principal=oryx/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.148.62:38971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:04.023949 14780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:04.024247 14780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:04.029901 14788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:04.029892 14785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:04.029889 14786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:04.029909 14780 server_base.cc:1034] running on GCE node
I20260812 14:23:04.030364 14780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:04.030658 14780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:04.031944 14780 hybrid_clock.cc:648] HybridClock initialized: now 1786544584031911 us; error 40 us; skew 500 ppm
Aug 12 14:23:04 dist-test-slave-8w9v krb5kdc[14478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544584, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:04.034734 14780 init.cc:377] Logged in from keytab as oryx/127.5.148.2@KRBTEST.COM (short username oryx)
I20260812 14:23:04.035238 14780 webserver.cc:460] Webserver started at http://127.5.148.2:41635/ using document root <none> and password file <none>
I20260812 14:23:04.035516 14780 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:04.035589 14780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:04.035743 14780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:04.036942 14780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "8b4957cf85ba48baae33b8cd4b1bd35a"
format_stamp: "Formatted at 2026-08-12 14:23:04 on dist-test-slave-8w9v"
server_key: "0176ac427b24836f15f58fc63dbd107d"
server_key_iv: "73481ce5df4163f41a49ca6359a87dad"
server_key_version: "encryptionkey@0"
I20260812 14:23:04.037322 14780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "8b4957cf85ba48baae33b8cd4b1bd35a"
format_stamp: "Formatted at 2026-08-12 14:23:04 on dist-test-slave-8w9v"
server_key: "0176ac427b24836f15f58fc63dbd107d"
server_key_iv: "73481ce5df4163f41a49ca6359a87dad"
server_key_version: "encryptionkey@0"
I20260812 14:23:04.038775 14780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:23:04.039614 14795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:04.039811 14780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:23:04.039933 14780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "8b4957cf85ba48baae33b8cd4b1bd35a"
format_stamp: "Formatted at 2026-08-12 14:23:04 on dist-test-slave-8w9v"
server_key: "0176ac427b24836f15f58fc63dbd107d"
server_key_iv: "73481ce5df4163f41a49ca6359a87dad"
server_key_version: "encryptionkey@0"
I20260812 14:23:04.040014 14780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:04.068378 14780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:04.071094 14780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:04.071271 14780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:04.071972 14780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:04.072028 14780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:04.072064 14780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:04.072083 14780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:23:04 dist-test-slave-8w9v krb5kdc[14478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544584, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
I20260812 14:23:04.080031 14780 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:45623
I20260812 14:23:04.080461 14780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:23:04.080456 14923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:45623 every 8 connection(s)
I20260812 14:23:04.082931 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.072520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:48473 (local address 127.5.148.62:38971)
0812 14:23:04.072601 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.072604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.072913 (+ 309us) server_negotiation.cc:407] Connection header received
0812 14:23:04.073685 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.073687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.073698 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.073805 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:04.074788 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.075290 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.076530 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.076733 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.076814 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.079696 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:04.079697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:04.079698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:04.079706 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:04.081414 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:04.081890 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:04.081892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:04.081892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:04.081952 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:04.082221 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:04.082222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:04.082223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:04.082352 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:04.082429 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.082728 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.082841 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 14:23:04.083106 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.072532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38971 (local address 127.5.148.2:48473)
0812 14:23:04.072719 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.072779 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.073558 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.073914 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.073918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.074338 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:04.074679 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.074685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.075448 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.075450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.076433 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.076436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.076832 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.076833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.076964 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.077580 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:04.077590 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:04.079592 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:04.081523 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:04.081533 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:04.081535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:04.081808 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:04.082037 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:04.082038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:04.082038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:04.082154 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:04.082430 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:04.082439 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:04.082587 (+ 148us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.082692 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.082864 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260812 14:23:04.083701 14924 heartbeater.cc:347] Connected to a master server at 127.5.148.62:38971
I20260812 14:23:04.083791 14924 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:04.084012 14924 heartbeater.cc:511] Master 127.5.148.62:38971 requested a full tablet report, sending...
I20260812 14:23:04.084448 14534 ts_manager.cc:194] Registered new tserver with Master: 8b4957cf85ba48baae33b8cd4b1bd35a (127.5.148.2:45623)
I20260812 14:23:04.084872 14534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:48473
I20260812 14:23:04.089072 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 14780
I20260812 14:23:04.089202 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:23:04.089478 5712 external_mini_cluster.cc:1428] Setting key 2b5c8668510ea9453fdfa5ec17973a57
I20260812 14:23:04.091107 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.084931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:55589 (local address 127.5.148.62:38971)
0812 14:23:04.085012 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.085014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.085169 (+ 155us) server_negotiation.cc:407] Connection header received
0812 14:23:04.085216 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.085217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.085228 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.085288 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:04.085708 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.086211 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.086993 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.087175 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.087232 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.088350 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:04.088352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:04.088352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:04.088357 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:04.089851 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:04.090268 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:04.090270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:04.090270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:04.090345 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:04.090600 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:04.090600 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:04.090601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:04.090748 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:04.090807 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.090977 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.091042 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 14:23:04.091344 14931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.084873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38971 (local address 127.5.148.2:55589)
0812 14:23:04.085083 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.085095 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.085156 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.085304 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.085305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.085514 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:04.085580 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.085583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.086349 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.086350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.086907 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.086909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.087254 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.087254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.087326 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.087869 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:04.087875 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:04.088268 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:04.089934 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:04.089935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:04.089935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:04.090178 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:04.090441 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:04.090442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:04.090443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:04.090528 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:04.090821 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:04.090822 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:04.090878 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.090961 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.091099 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for oryx/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.3.keytab.
Entry for principal oryx/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.3.keytab.
I20260812 14:23:04.120319 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:38971
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.3.keytab
--principal=oryx/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:23:04.207275 14935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:04.207532 14935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:04.207577 14935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:04.208973 14935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:04.209064 14935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:04.209102 14935 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:23:04.209142 14935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:23:04.210784 14935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.3.keytab
--principal=oryx/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.148.62:38971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:04.211102 14935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:04.211357 14935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:04.216873 14943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:04.216849 14940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:04.216951 14941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:04.217185 14935 server_base.cc:1034] running on GCE node
I20260812 14:23:04.217398 14935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:04.217617 14935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:04.218806 14935 hybrid_clock.cc:648] HybridClock initialized: now 1786544584218789 us; error 35 us; skew 500 ppm
Aug 12 14:23:04 dist-test-slave-8w9v krb5kdc[14478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544584, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:04.221771 14935 init.cc:377] Logged in from keytab as oryx/127.5.148.3@KRBTEST.COM (short username oryx)
I20260812 14:23:04.222234 14935 webserver.cc:460] Webserver started at http://127.5.148.3:44055/ using document root <none> and password file <none>
I20260812 14:23:04.222498 14935 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:04.222548 14935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:04.222718 14935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:04.223810 14935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "8caaa6377b6c4d1790191186b6715c90"
format_stamp: "Formatted at 2026-08-12 14:23:04 on dist-test-slave-8w9v"
server_key: "9819649e2f912793962f945ee8e8d59a"
server_key_iv: "de205b5d4d08374c00ff9960e161b64c"
server_key_version: "encryptionkey@0"
I20260812 14:23:04.224263 14935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "8caaa6377b6c4d1790191186b6715c90"
format_stamp: "Formatted at 2026-08-12 14:23:04 on dist-test-slave-8w9v"
server_key: "9819649e2f912793962f945ee8e8d59a"
server_key_iv: "de205b5d4d08374c00ff9960e161b64c"
server_key_version: "encryptionkey@0"
I20260812 14:23:04.225656 14935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:04.226432 14951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:04.226624 14935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:23:04.226718 14935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "8caaa6377b6c4d1790191186b6715c90"
format_stamp: "Formatted at 2026-08-12 14:23:04 on dist-test-slave-8w9v"
server_key: "9819649e2f912793962f945ee8e8d59a"
server_key_iv: "de205b5d4d08374c00ff9960e161b64c"
server_key_version: "encryptionkey@0"
I20260812 14:23:04.226797 14935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:04.234968 14935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:04.237303 14935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:04.237459 14935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:04.238039 14935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:04.238078 14935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:04.238103 14935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:04.238117 14935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:23:04 dist-test-slave-8w9v krb5kdc[14478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544584, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
I20260812 14:23:04.245795 14935 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:40879
I20260812 14:23:04.245851 15080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:40879 every 8 connection(s)
I20260812 14:23:04.246234 14935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:23:04.249017 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.238676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:33657 (local address 127.5.148.62:38971)
0812 14:23:04.238757 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.238761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.238987 (+ 226us) server_negotiation.cc:407] Connection header received
0812 14:23:04.239570 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.239572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.239587 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.239674 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:04.240551 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.241343 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.243004 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.243339 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.243410 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.246105 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:04.246107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:04.246109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:04.246122 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:04.247599 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:04.247986 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:04.247987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:04.247987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:04.248043 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:04.248341 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:04.248341 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:04.248342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:04.248493 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:04.248553 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.248877 (+ 324us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.248951 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 14:23:04.249308 14961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.238641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38971 (local address 127.5.148.3:33657)
0812 14:23:04.238832 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.238877 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.239416 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.239770 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.239775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.240194 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:04.240428 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.240434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.241449 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.241450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.242909 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.242913 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.243439 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.243440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.243606 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.244276 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:04.244286 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:04.245993 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:04.247679 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:04.247687 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:04.247688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:04.247913 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:04.248104 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:04.248105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:04.248105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:04.248279 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:04.248586 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:04.248595 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:04.248733 (+ 138us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.248839 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.248982 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
I20260812 14:23:04.249868 15082 heartbeater.cc:347] Connected to a master server at 127.5.148.62:38971
I20260812 14:23:04.249971 15082 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:04.250175 15082 heartbeater.cc:511] Master 127.5.148.62:38971 requested a full tablet report, sending...
I20260812 14:23:04.250623 14536 ts_manager.cc:194] Registered new tserver with Master: 8caaa6377b6c4d1790191186b6715c90 (127.5.148.3:40879)
I20260812 14:23:04.251143 14536 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:33657
I20260812 14:23:04.256327 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 14935
I20260812 14:23:04.256434 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:23:04.256635 5712 external_mini_cluster.cc:1428] Setting key b2334eb405bb0db9bc05be74c2c2ffb0
I20260812 14:23:04.257081 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.251047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38971 (local address 127.5.148.3:44547)
0812 14:23:04.251193 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.251205 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.251297 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.251430 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.251431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.251643 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:04.251723 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.251725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.252427 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.252428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.252927 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.252928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.253235 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.253236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.253284 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.253691 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:04.253695 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:04.253986 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:04.255503 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:04.255505 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:04.255505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:04.255663 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:04.255860 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:04.255862 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:04.255862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:04.255935 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:04.256741 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:04.256742 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:04.256801 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.256903 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.257000 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260812 14:23:04.257148 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.251144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:44547 (local address 127.5.148.62:38971)
0812 14:23:04.251187 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.251188 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.251270 (+ 82us) server_negotiation.cc:407] Connection header received
0812 14:23:04.251350 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.251351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.251361 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.251424 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:04.251804 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.252330 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.252997 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.253170 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.253220 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.254062 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:04.254063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:04.254063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:04.254069 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:04.255432 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:04.255726 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:04.255726 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:04.255727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:04.255775 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:04.256020 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:04.256020 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:04.256021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:04.256159 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:04.256221 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.256954 (+ 733us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.257044 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:23:04.257987 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 14:23:04 dist-test-slave-8w9v krb5kdc[14478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 14:23:04 dist-test-slave-8w9v krb5kdc[14478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:04 dist-test-slave-8w9v krb5kdc[14478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 14:23:04.283216 14594 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:04.277797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.5.148.62:38971)
0812 14:23:04.277865 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.277867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.277886 (+ 19us) server_negotiation.cc:407] Connection header received
0812 14:23:04.277949 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.277950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.277960 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.278014 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:04.278597 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.279092 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.279819 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.280002 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.280053 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.283115 (+ 3062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58456: BlockingRecv error: recv got EOF from 127.0.0.1:58456 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 12 14:23:04 dist-test-slave-8w9v krb5kdc[14478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
I20260812 14:23:04.291633 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.284097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58464 (local address 127.5.148.62:38971)
0812 14:23:04.284188 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.284191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.284282 (+ 91us) server_negotiation.cc:407] Connection header received
0812 14:23:04.284356 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.284357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.284368 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.284422 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:04.284942 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.285435 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.286133 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.286331 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.286410 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.288675 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:04.288676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:04.288676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:04.288683 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:04.290167 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:04.290844 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:04.290845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:04.290846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:04.290907 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:04.291143 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:04.291144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:04.291144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:04.291262 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:04.291343 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.291500 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.291564 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 14:23:04.293443 14536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:23:04.295141 14536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:23:04.302495 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.297948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.5.148.3:40879)
0812 14:23:04.298220 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.298225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.298282 (+ 57us) server_negotiation.cc:407] Connection header received
0812 14:23:04.298376 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.298379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.298398 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.298487 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:04.298783 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.299969 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.301253 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.302062 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.302145 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.302235 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.302343 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.302393 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
I20260812 14:23:04.302476 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.297809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:40879 (local address 127.0.0.1:54456)
0812 14:23:04.298216 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.298229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.298294 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.298519 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.298523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.298525 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:04.298606 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.298608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.300076 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.300076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.301156 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.301159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.302151 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.302152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.302213 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.302225 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.302319 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.302383 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":28,"thread_start_us":67,"threads_started":1}
I20260812 14:23:04.303143 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.297808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:41081 (local address 127.0.0.1:53672)
0812 14:23:04.298007 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.298051 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.298126 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.298560 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.298561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.298564 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:04.298626 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.298628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.300020 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.300021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.301355 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.301359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.302302 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.302303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.302358 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.302362 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.302443 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.302477 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":40,"thread_start_us":138,"threads_started":1}
I20260812 14:23:04.303515 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.298184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.5.148.1:41081)
0812 14:23:04.298389 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.298394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.298403 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:23:04.298442 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.298444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.298458 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.298706 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:04.298773 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.299944 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.301485 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.302216 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.302287 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.302373 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.303305 (+ 932us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.303365 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260812 14:23:04.303378 15010 tablet_service.cc:1467] Processing CreateTablet for tablet 22625999900a4af78fd7bf7325b07dd4 (DEFAULT_TABLE table=test-table [id=4c9690ba103d488a8e2971d8a04f1426]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:04.303671 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.298070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:45623 (local address 127.0.0.1:57658)
0812 14:23:04.298608 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.298619 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.298662 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.298976 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.298976 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.298978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:04.299007 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.299008 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.299935 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.299936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.301356 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.301359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.303408 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.303410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.303477 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.303481 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.303542 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.303591 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":74,"thread_start_us":93,"threads_started":1}
I20260812 14:23:04.303809 15010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22625999900a4af78fd7bf7325b07dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:04.304039 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.298386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57658 (local address 127.5.148.2:45623)
0812 14:23:04.298745 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.298751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.298762 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:23:04.298801 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.298803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.298820 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.298894 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:04.299086 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.299834 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.302434 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.303208 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.303283 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.303375 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.303903 (+ 528us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.303951 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":124,"threads_started":1}
I20260812 14:23:04.304235 14692 tablet_service.cc:1467] Processing CreateTablet for tablet 22625999900a4af78fd7bf7325b07dd4 (DEFAULT_TABLE table=test-table [id=4c9690ba103d488a8e2971d8a04f1426]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:04.304605 14692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22625999900a4af78fd7bf7325b07dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:04.308034 15115 tablet_bootstrap.cc:492] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90: Bootstrap starting.
I20260812 14:23:04.308642 15116 tablet_bootstrap.cc:492] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a: Bootstrap starting.
I20260812 14:23:04.308773 15115 tablet_bootstrap.cc:654] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:04.308840 14852 tablet_service.cc:1467] Processing CreateTablet for tablet 22625999900a4af78fd7bf7325b07dd4 (DEFAULT_TABLE table=test-table [id=4c9690ba103d488a8e2971d8a04f1426]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:04.309182 14852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22625999900a4af78fd7bf7325b07dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:04.309250 15116 tablet_bootstrap.cc:654] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:04.309357 15115 log.cc:824] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90: Log is configured to *not* fsync() on all Append() calls
I20260812 14:23:04.309629 15116 log.cc:824] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:23:04.310217 15115 tablet_bootstrap.cc:492] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90: No bootstrap required, opened a new log
I20260812 14:23:04.310293 15115 ts_tablet_manager.cc:1397] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:23:04.312517 15115 raft_consensus.cc:348] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } }
I20260812 14:23:04.312654 15116 tablet_bootstrap.cc:492] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a: No bootstrap required, opened a new log
I20260812 14:23:04.312698 15115 raft_consensus.cc:374] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:04.312742 15116 ts_tablet_manager.cc:1397] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260812 14:23:04.312744 15115 raft_consensus.cc:729] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8caaa6377b6c4d1790191186b6715c90, State: Initialized, Role: FOLLOWER
I20260812 14:23:04.312904 15115 consensus_queue.cc:260] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } }
I20260812 14:23:04.313187 15115 ts_tablet_manager.cc:1428] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 14:23:04.313313 15082 heartbeater.cc:503] Master 127.5.148.62:38971 was elected leader, sending a full tablet report...
I20260812 14:23:04.313475 15119 tablet_bootstrap.cc:492] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a: Bootstrap starting.
I20260812 14:23:04.314172 15119 tablet_bootstrap.cc:654] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:04.314512 15119 log.cc:824] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:23:04.315093 15116 raft_consensus.cc:348] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } }
I20260812 14:23:04.315263 15116 raft_consensus.cc:374] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:04.315284 15119 tablet_bootstrap.cc:492] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a: No bootstrap required, opened a new log
I20260812 14:23:04.315302 15116 raft_consensus.cc:729] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4ce81490bf41de8b8f6747be5e945a, State: Initialized, Role: FOLLOWER
I20260812 14:23:04.315340 15119 ts_tablet_manager.cc:1397] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:23:04.315443 15116 consensus_queue.cc:260] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } }
I20260812 14:23:04.315735 15116 ts_tablet_manager.cc:1428] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:23:04.315788 14771 heartbeater.cc:503] Master 127.5.148.62:38971 was elected leader, sending a full tablet report...
I20260812 14:23:04.317014 15119 raft_consensus.cc:348] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } }
I20260812 14:23:04.317137 15119 raft_consensus.cc:374] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:04.317158 15119 raft_consensus.cc:729] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4957cf85ba48baae33b8cd4b1bd35a, State: Initialized, Role: FOLLOWER
I20260812 14:23:04.317296 15119 consensus_queue.cc:260] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } }
I20260812 14:23:04.317575 15119 ts_tablet_manager.cc:1428] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:23:04.317622 14924 heartbeater.cc:503] Master 127.5.148.62:38971 was elected leader, sending a full tablet report...
W20260812 14:23:04.331851 14926 tablet.cc:2369] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:23:04.381834 15123 raft_consensus.cc:482] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:23:04.381979 15123 raft_consensus.cc:504] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } }
I20260812 14:23:04.382351 15123 leader_election.cc:288] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8caaa6377b6c4d1790191186b6715c90 (127.5.148.3:40879), ce4ce81490bf41de8b8f6747be5e945a (127.5.148.1:41081)
I20260812 14:23:04.385681 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.382648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:46981 (local address 127.5.148.3:40879)
0812 14:23:04.382697 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.382699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.382735 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:23:04.382764 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.382765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.382773 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.382815 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:04.383045 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.383522 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.384753 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.385317 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.385384 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.385397 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.385593 (+ 196us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.385626 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:23:04.385702 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.382555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:40879 (local address 127.5.148.2:46981)
0812 14:23:04.382627 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.382639 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.382692 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.382850 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.382850 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.382852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:04.382953 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.382955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.383599 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.383600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.384667 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.384670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.385419 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.385420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.385510 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.385514 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.385578 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.385620 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":1}
I20260812 14:23:04.386051 15030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22625999900a4af78fd7bf7325b07dd4" candidate_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8caaa6377b6c4d1790191186b6715c90" is_pre_election: true
I20260812 14:23:04.386245 15030 raft_consensus.cc:2454] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b4957cf85ba48baae33b8cd4b1bd35a in term 0.
I20260812 14:23:04.386399 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.382560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:41081 (local address 127.5.148.2:51305)
0812 14:23:04.383079 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.383087 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.383151 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.383306 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.383307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.383309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:04.383381 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.383383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.384305 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.384306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.385300 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.385302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.386059 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.386060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.386141 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.386145 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.386227 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.386266 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":311,"thread_start_us":141,"threads_started":1}
I20260812 14:23:04.386451 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.382627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:51305 (local address 127.5.148.1:41081)
0812 14:23:04.382684 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.382687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.383141 (+ 454us) server_negotiation.cc:407] Connection header received
0812 14:23:04.383216 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.383217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.383237 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.383300 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:04.383460 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.384224 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.385420 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.385959 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.386029 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.386038 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.386309 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.386368 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 14:23:04.386523 14797 leader_election.cc:302] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b4957cf85ba48baae33b8cd4b1bd35a, 8caaa6377b6c4d1790191186b6715c90; no voters:
I20260812 14:23:04.386656 15123 raft_consensus.cc:2781] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:23:04.386723 15123 raft_consensus.cc:482] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:23:04.386755 15123 raft_consensus.cc:3037] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:04.386937 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22625999900a4af78fd7bf7325b07dd4" candidate_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4ce81490bf41de8b8f6747be5e945a" is_pre_election: true
I20260812 14:23:04.387091 14716 raft_consensus.cc:2454] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b4957cf85ba48baae33b8cd4b1bd35a in term 0.
I20260812 14:23:04.387871 15123 raft_consensus.cc:504] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } }
I20260812 14:23:04.388029 15123 leader_election.cc:288] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [CANDIDATE]: Term 1 election: Requested vote from peers 8caaa6377b6c4d1790191186b6715c90 (127.5.148.3:40879), ce4ce81490bf41de8b8f6747be5e945a (127.5.148.1:41081)
I20260812 14:23:04.388212 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22625999900a4af78fd7bf7325b07dd4" candidate_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4ce81490bf41de8b8f6747be5e945a"
I20260812 14:23:04.388229 15030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22625999900a4af78fd7bf7325b07dd4" candidate_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8caaa6377b6c4d1790191186b6715c90"
I20260812 14:23:04.388310 15030 raft_consensus.cc:3037] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:04.388311 14716 raft_consensus.cc:3037] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:04.389168 15030 raft_consensus.cc:2454] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b4957cf85ba48baae33b8cd4b1bd35a in term 1.
I20260812 14:23:04.389196 14716 raft_consensus.cc:2454] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b4957cf85ba48baae33b8cd4b1bd35a in term 1.
I20260812 14:23:04.389344 14797 leader_election.cc:302] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b4957cf85ba48baae33b8cd4b1bd35a, 8caaa6377b6c4d1790191186b6715c90; no voters:
I20260812 14:23:04.389465 15123 raft_consensus.cc:2781] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:23:04.389624 15123 raft_consensus.cc:686] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 LEADER]: Becoming Leader. State: Replica: 8b4957cf85ba48baae33b8cd4b1bd35a, State: Running, Role: LEADER
I20260812 14:23:04.389739 15123 consensus_queue.cc:237] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } }
I20260812 14:23:04.390645 14536 catalog_manager.cc:5495] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b4957cf85ba48baae33b8cd4b1bd35a (127.5.148.2). New cstate: current_term: 1 leader_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } health_report { overall_health: HEALTHY } } }
W20260812 14:23:04.394939 14772 tablet.cc:2369] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:23:04.417569 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.414963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57660 (local address 127.5.148.2:45623)
0812 14:23:04.415030 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.415033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.415042 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:23:04.415093 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.415094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.415115 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.415208 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:23:04.415369 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.415870 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.416751 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.416956 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.417031 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.417222 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:23:04.417332 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:23:04.417431 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.417451 (+ 20us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.417474 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:23:04.419610 14716 raft_consensus.cc:1261] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Refusing update from remote peer 8b4957cf85ba48baae33b8cd4b1bd35a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:04.419637 15030 raft_consensus.cc:1261] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Refusing update from remote peer 8b4957cf85ba48baae33b8cd4b1bd35a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:04.419992 15123 consensus_queue.cc:1035] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:04.420202 15125 consensus_queue.cc:1035] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:04.423206 15132 mvcc.cc:205] Tried to move back new op lower bound from 7317686617779707904 to 7317686617661136896. Current Snapshot: MvccSnapshot[applied={T|T < 7317686617779707904}]
I20260812 14:23:04.426287 14536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:23:04.426376 14536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:23:04.427196 14536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb: Sending DeleteTablet for 3 replicas of tablet 22625999900a4af78fd7bf7325b07dd4
I20260812 14:23:04.427556 15010 tablet_service.cc:1514] Processing DeleteTablet for tablet 22625999900a4af78fd7bf7325b07dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:04 UTC) from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:54456
I20260812 14:23:04.427553 14852 tablet_service.cc:1514] Processing DeleteTablet for tablet 22625999900a4af78fd7bf7325b07dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:04 UTC) from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:57658
I20260812 14:23:04.427568 14692 tablet_service.cc:1514] Processing DeleteTablet for tablet 22625999900a4af78fd7bf7325b07dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:04 UTC) from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:53672
I20260812 14:23:04.427816 14536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:23:04.428004 14536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:23:04.428082 15142 tablet_replica.cc:330] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90: stopping tablet replica
I20260812 14:23:04.428275 15142 raft_consensus.cc:2229] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:23:04.428496 15142 raft_consensus.cc:2258] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:04.428665 15141 tablet_replica.cc:330] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a: stopping tablet replica
I20260812 14:23:04.428809 15141 raft_consensus.cc:2229] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:23:04.428972 15141 raft_consensus.cc:2258] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:04.429394 14692 tablet_service.cc:1467] Processing CreateTablet for tablet da3e8def1c15412688510f77573ecade (DEFAULT_TABLE table=test-table [id=c33731eefa4547dcbb4561674c4a4508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:04.429545 14692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da3e8def1c15412688510f77573ecade. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:04.429553 14852 tablet_service.cc:1467] Processing CreateTablet for tablet da3e8def1c15412688510f77573ecade (DEFAULT_TABLE table=test-table [id=c33731eefa4547dcbb4561674c4a4508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:04.429682 14852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da3e8def1c15412688510f77573ecade. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:04.430655 15116 tablet_bootstrap.cc:492] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a: Bootstrap starting.
I20260812 14:23:04.431092 15116 tablet_bootstrap.cc:654] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:04.431187 15010 tablet_service.cc:1467] Processing CreateTablet for tablet da3e8def1c15412688510f77573ecade (DEFAULT_TABLE table=test-table [id=c33731eefa4547dcbb4561674c4a4508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:04.431299 15010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da3e8def1c15412688510f77573ecade. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:04.431421 15119 tablet_bootstrap.cc:492] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a: Bootstrap starting.
I20260812 14:23:04.431427 15142 ts_tablet_manager.cc:1905] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:04.431845 15119 tablet_bootstrap.cc:654] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:04.432552 15116 tablet_bootstrap.cc:492] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a: No bootstrap required, opened a new log
I20260812 14:23:04.432602 15116 ts_tablet_manager.cc:1397] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:23:04.432701 15115 tablet_bootstrap.cc:492] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90: Bootstrap starting.
I20260812 14:23:04.432770 15116 raft_consensus.cc:348] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.432847 15116 raft_consensus.cc:374] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:04.432873 15116 raft_consensus.cc:729] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4ce81490bf41de8b8f6747be5e945a, State: Initialized, Role: FOLLOWER
I20260812 14:23:04.432932 15116 consensus_queue.cc:260] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.433060 15116 ts_tablet_manager.cc:1428] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:23:04.433774 15142 ts_tablet_manager.cc:1918] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:23:04.433880 15142 log.cc:1196] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/22625999900a4af78fd7bf7325b07dd4
I20260812 14:23:04.434561 15143 tablet_replica.cc:330] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a: stopping tablet replica
I20260812 14:23:04.434690 15143 raft_consensus.cc:2229] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:23:04.434717 15115 tablet_bootstrap.cc:654] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:04.434795 15142 ts_tablet_manager.cc:1939] T 22625999900a4af78fd7bf7325b07dd4 P 8caaa6377b6c4d1790191186b6715c90: Deleting consensus metadata
I20260812 14:23:04.434873 15143 raft_consensus.cc:2258] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:04.435281 14515 catalog_manager.cc:4841] TS 8caaa6377b6c4d1790191186b6715c90 (127.5.148.3:40879): tablet 22625999900a4af78fd7bf7325b07dd4 (table test-table [id=4c9690ba103d488a8e2971d8a04f1426]) successfully deleted
I20260812 14:23:04.435322 15141 ts_tablet_manager.cc:1905] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:04.436467 15143 ts_tablet_manager.cc:1905] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:04.437126 15141 ts_tablet_manager.cc:1918] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:23:04.437212 15141 log.cc:1196] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/wal/wals/22625999900a4af78fd7bf7325b07dd4
I20260812 14:23:04.437904 15141 ts_tablet_manager.cc:1939] T 22625999900a4af78fd7bf7325b07dd4 P ce4ce81490bf41de8b8f6747be5e945a: Deleting consensus metadata
I20260812 14:23:04.438333 15143 ts_tablet_manager.cc:1918] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:23:04.438413 15143 log.cc:1196] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/22625999900a4af78fd7bf7325b07dd4
I20260812 14:23:04.438390 14514 catalog_manager.cc:4841] TS ce4ce81490bf41de8b8f6747be5e945a (127.5.148.1:41081): tablet 22625999900a4af78fd7bf7325b07dd4 (table test-table [id=4c9690ba103d488a8e2971d8a04f1426]) successfully deleted
I20260812 14:23:04.438620 15115 tablet_bootstrap.cc:492] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90: No bootstrap required, opened a new log
I20260812 14:23:04.438684 15115 ts_tablet_manager.cc:1397] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.000s
I20260812 14:23:04.438710 15143 ts_tablet_manager.cc:1939] T 22625999900a4af78fd7bf7325b07dd4 P 8b4957cf85ba48baae33b8cd4b1bd35a: Deleting consensus metadata
I20260812 14:23:04.438850 15115 raft_consensus.cc:348] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.438935 15115 raft_consensus.cc:374] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:04.438966 15115 raft_consensus.cc:729] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8caaa6377b6c4d1790191186b6715c90, State: Initialized, Role: FOLLOWER
I20260812 14:23:04.439015 15115 consensus_queue.cc:260] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.439117 15115 ts_tablet_manager.cc:1428] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:04.439224 14515 catalog_manager.cc:4841] TS 8b4957cf85ba48baae33b8cd4b1bd35a (127.5.148.2:45623): tablet 22625999900a4af78fd7bf7325b07dd4 (table test-table [id=4c9690ba103d488a8e2971d8a04f1426]) successfully deleted
I20260812 14:23:04.439538 15119 tablet_bootstrap.cc:492] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a: No bootstrap required, opened a new log
I20260812 14:23:04.439587 15119 ts_tablet_manager.cc:1397] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260812 14:23:04.439787 15119 raft_consensus.cc:348] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.439877 15119 raft_consensus.cc:374] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:04.439900 15119 raft_consensus.cc:729] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4957cf85ba48baae33b8cd4b1bd35a, State: Initialized, Role: FOLLOWER
I20260812 14:23:04.439944 15119 consensus_queue.cc:260] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.440034 15119 ts_tablet_manager.cc:1428] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260812 14:23:04.497067 15083 tablet.cc:2369] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:23:04.630851 15122 raft_consensus.cc:482] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:23:04.630956 15122 raft_consensus.cc:504] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.631320 15122 leader_election.cc:288] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b4957cf85ba48baae33b8cd4b1bd35a (127.5.148.2:45623), 8caaa6377b6c4d1790191186b6715c90 (127.5.148.3:40879)
I20260812 14:23:04.634680 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.631868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:49461 (local address 127.5.148.3:40879)
0812 14:23:04.631949 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.631951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.631983 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:23:04.632086 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.632087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.632096 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.632196 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:04.632365 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.632838 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.633868 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.634246 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.634344 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.634355 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.634538 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.634577 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 14:23:04.634704 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.631524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:45623 (local address 127.5.148.1:33941)
0812 14:23:04.631696 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.631709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.631812 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.632036 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.632037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.632040 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:04.632183 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.632186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.632883 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.632884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.633725 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.633728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.634396 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.634397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.634460 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.634464 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.634506 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.634541 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260812 14:23:04.634769 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.631627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:33941 (local address 127.5.148.2:45623)
0812 14:23:04.631704 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.631707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.631790 (+ 83us) server_negotiation.cc:407] Connection header received
0812 14:23:04.631948 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.631949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.631966 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.632030 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:04.632276 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.632785 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.633825 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.634255 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.634338 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.634357 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.634643 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.634695 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 14:23:04.634714 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.631749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:40879 (local address 127.5.148.1:49461)
0812 14:23:04.631904 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.631926 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.631999 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.632201 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.632202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.632204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:04.632289 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.632291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.632948 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.632949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.633772 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.633775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.634377 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.634378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.634459 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.634463 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.634531 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.634577 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":24,"thread_start_us":85,"threads_started":1}
I20260812 14:23:04.635085 14875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da3e8def1c15412688510f77573ecade" candidate_uuid: "ce4ce81490bf41de8b8f6747be5e945a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" is_pre_election: true
I20260812 14:23:04.635259 14875 raft_consensus.cc:2454] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce4ce81490bf41de8b8f6747be5e945a in term 0.
I20260812 14:23:04.635280 15030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da3e8def1c15412688510f77573ecade" candidate_uuid: "ce4ce81490bf41de8b8f6747be5e945a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8caaa6377b6c4d1790191186b6715c90" is_pre_election: true
I20260812 14:23:04.635385 15030 raft_consensus.cc:2454] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce4ce81490bf41de8b8f6747be5e945a in term 0.
I20260812 14:23:04.635488 14639 leader_election.cc:302] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b4957cf85ba48baae33b8cd4b1bd35a, ce4ce81490bf41de8b8f6747be5e945a; no voters:
I20260812 14:23:04.635640 15122 raft_consensus.cc:2781] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:23:04.635675 15122 raft_consensus.cc:482] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:23:04.635690 15122 raft_consensus.cc:3037] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:04.636569 15122 raft_consensus.cc:504] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.636711 15122 leader_election.cc:288] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [CANDIDATE]: Term 1 election: Requested vote from peers 8b4957cf85ba48baae33b8cd4b1bd35a (127.5.148.2:45623), 8caaa6377b6c4d1790191186b6715c90 (127.5.148.3:40879)
I20260812 14:23:04.636885 14875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da3e8def1c15412688510f77573ecade" candidate_uuid: "ce4ce81490bf41de8b8f6747be5e945a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a"
I20260812 14:23:04.636901 15030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da3e8def1c15412688510f77573ecade" candidate_uuid: "ce4ce81490bf41de8b8f6747be5e945a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8caaa6377b6c4d1790191186b6715c90"
I20260812 14:23:04.636969 14875 raft_consensus.cc:3037] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:04.636991 15030 raft_consensus.cc:3037] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:04.637678 14875 raft_consensus.cc:2454] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce4ce81490bf41de8b8f6747be5e945a in term 1.
I20260812 14:23:04.637679 15030 raft_consensus.cc:2454] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce4ce81490bf41de8b8f6747be5e945a in term 1.
I20260812 14:23:04.637869 14639 leader_election.cc:302] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8caaa6377b6c4d1790191186b6715c90, ce4ce81490bf41de8b8f6747be5e945a; no voters:
I20260812 14:23:04.637975 15122 raft_consensus.cc:2781] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:23:04.638118 15122 raft_consensus.cc:686] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [term 1 LEADER]: Becoming Leader. State: Replica: ce4ce81490bf41de8b8f6747be5e945a, State: Running, Role: LEADER
I20260812 14:23:04.638175 15122 consensus_queue.cc:237] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.639024 14536 catalog_manager.cc:5495] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a reported cstate change: term changed from 0 to 1, leader changed from <none> to ce4ce81490bf41de8b8f6747be5e945a (127.5.148.1). New cstate: current_term: 1 leader_uuid: "ce4ce81490bf41de8b8f6747be5e945a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } health_report { overall_health: UNKNOWN } } }
Aug 12 14:23:04 dist-test-slave-8w9v krb5kdc[14478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544583, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.62@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
I20260812 14:23:04.692268 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.685026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.5.148.62:38971)
0812 14:23:04.685106 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.685109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.685226 (+ 117us) server_negotiation.cc:407] Connection header received
0812 14:23:04.685269 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.685270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.685279 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.685332 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:04.685826 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.686320 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.687025 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.687199 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.687251 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.689485 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:04.689486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:04.689486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:04.689493 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:04.690945 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:04.691436 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:04.691437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:04.691437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:04.691491 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:04.691688 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:04.691688 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:04.691689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:04.691791 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:04.691841 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.692010 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.692197 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":47}
I20260812 14:23:04.692351 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.684949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:38971 (local address 127.0.0.1:58466)
0812 14:23:04.685138 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.685148 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.685222 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.685377 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.685378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.685699 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:04.685755 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.685757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.686417 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.686418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.686951 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.686952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.687269 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.687270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.687322 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.687852 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:04.687857 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:04.689392 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:04.691045 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:04.691054 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:04.691056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:04.691354 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:04.691556 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:04.691557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:04.691557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:04.691629 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:04.691853 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:04.691861 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:04.691909 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.691978 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.692088 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":109,"threads_started":1}
I20260812 14:23:04.694655 14536 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:58466:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 14:23:04.694916 14536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:23:04.696770 14692 tablet_service.cc:1467] Processing CreateTablet for tablet 7d278eade1ec4b388b42fb657fa08865 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0965d8ed8b71428292d946da4a128cad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:23:04.696771 15010 tablet_service.cc:1467] Processing CreateTablet for tablet 7d278eade1ec4b388b42fb657fa08865 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0965d8ed8b71428292d946da4a128cad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:23:04.696771 14852 tablet_service.cc:1467] Processing CreateTablet for tablet 7d278eade1ec4b388b42fb657fa08865 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0965d8ed8b71428292d946da4a128cad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:23:04.696957 14692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d278eade1ec4b388b42fb657fa08865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:04.696957 15010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d278eade1ec4b388b42fb657fa08865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:04.696978 14852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d278eade1ec4b388b42fb657fa08865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:04.698382 15119 tablet_bootstrap.cc:492] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a: Bootstrap starting.
I20260812 14:23:04.698611 15115 tablet_bootstrap.cc:492] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90: Bootstrap starting.
I20260812 14:23:04.698635 15116 tablet_bootstrap.cc:492] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a: Bootstrap starting.
I20260812 14:23:04.698787 15119 tablet_bootstrap.cc:654] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:04.699060 15115 tablet_bootstrap.cc:654] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:04.699043 15116 tablet_bootstrap.cc:654] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:04.699491 15119 tablet_bootstrap.cc:492] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a: No bootstrap required, opened a new log
I20260812 14:23:04.699546 15119 ts_tablet_manager.cc:1397] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:04.699713 15119 raft_consensus.cc:348] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.699820 15119 raft_consensus.cc:374] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:04.699851 15119 raft_consensus.cc:729] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4957cf85ba48baae33b8cd4b1bd35a, State: Initialized, Role: FOLLOWER
I20260812 14:23:04.699838 15115 tablet_bootstrap.cc:492] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90: No bootstrap required, opened a new log
I20260812 14:23:04.699895 15115 ts_tablet_manager.cc:1397] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260812 14:23:04.699911 15119 consensus_queue.cc:260] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.700032 15119 ts_tablet_manager.cc:1428] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:04.700057 15125 tablet_replica.cc:439] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } }
I20260812 14:23:04.700048 15115 raft_consensus.cc:348] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.700181 15115 raft_consensus.cc:374] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:04.700171 15125 tablet_replica.cc:442] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:23:04.700206 15115 raft_consensus.cc:729] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8caaa6377b6c4d1790191186b6715c90, State: Initialized, Role: FOLLOWER
I20260812 14:23:04.700251 15115 consensus_queue.cc:260] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.700338 15115 ts_tablet_manager.cc:1428] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:04.700448 15120 tablet_replica.cc:439] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } }
I20260812 14:23:04.700568 15120 tablet_replica.cc:442] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:23:04.700820 15116 tablet_bootstrap.cc:492] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a: No bootstrap required, opened a new log
I20260812 14:23:04.700878 15116 ts_tablet_manager.cc:1397] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:23:04.701043 15116 raft_consensus.cc:348] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.701133 15116 raft_consensus.cc:374] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:04.701152 15116 raft_consensus.cc:729] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4ce81490bf41de8b8f6747be5e945a, State: Initialized, Role: FOLLOWER
I20260812 14:23:04.701220 15116 consensus_queue.cc:260] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.701318 15116 ts_tablet_manager.cc:1428] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:04.701416 15122 tablet_replica.cc:439] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } }
I20260812 14:23:04.701551 15122 tablet_replica.cc:442] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:23:04.828923 15120 raft_consensus.cc:482] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:23:04.829027 15120 raft_consensus.cc:504] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.829504 15120 leader_election.cc:288] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce4ce81490bf41de8b8f6747be5e945a (127.5.148.1:41081), 8b4957cf85ba48baae33b8cd4b1bd35a (127.5.148.2:45623)
I20260812 14:23:04.833810 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.829808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:42485 (local address 127.5.148.1:41081)
0812 14:23:04.829874 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.829876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.830093 (+ 217us) server_negotiation.cc:407] Connection header received
0812 14:23:04.830122 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.830123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.830132 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.830174 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:04.830380 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.831189 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.832833 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.833423 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.833487 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.833497 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.833690 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.833737 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 14:23:04.833810 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.829893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:35049 (local address 127.5.148.2:45623)
0812 14:23:04.829940 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.829942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.829982 (+ 40us) server_negotiation.cc:407] Connection header received
0812 14:23:04.830228 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.830229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.830239 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.830280 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:04.830451 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.831198 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.832833 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.833390 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.833478 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.833491 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.833690 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.833752 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:23:04.833853 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.829710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:41081 (local address 127.5.148.3:42485)
0812 14:23:04.829988 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.829999 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.830055 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.830197 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.830197 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.830199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:04.830296 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.830298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.831276 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.831278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.832727 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.832732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.833502 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.833504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.833601 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.833608 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.833669 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.833726 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":112,"thread_start_us":54,"threads_started":1}
I20260812 14:23:04.834113 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.829710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:45623 (local address 127.5.148.3:35049)
0812 14:23:04.829885 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.829898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.830007 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.830304 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.830305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.830307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:04.830363 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.830364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.831279 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.831279 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.832730 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.832732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.833504 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.833504 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.833601 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.833610 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.833669 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.833726 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":87,"thread_start_us":76,"threads_started":1}
I20260812 14:23:04.834388 14875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d278eade1ec4b388b42fb657fa08865" candidate_uuid: "8caaa6377b6c4d1790191186b6715c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" is_pre_election: true
I20260812 14:23:04.834438 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d278eade1ec4b388b42fb657fa08865" candidate_uuid: "8caaa6377b6c4d1790191186b6715c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4ce81490bf41de8b8f6747be5e945a" is_pre_election: true
I20260812 14:23:04.834506 14875 raft_consensus.cc:2454] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8caaa6377b6c4d1790191186b6715c90 in term 0.
I20260812 14:23:04.834518 14716 raft_consensus.cc:2454] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8caaa6377b6c4d1790191186b6715c90 in term 0.
I20260812 14:23:04.834712 14952 leader_election.cc:302] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8caaa6377b6c4d1790191186b6715c90, ce4ce81490bf41de8b8f6747be5e945a; no voters:
I20260812 14:23:04.834872 15120 raft_consensus.cc:2781] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:23:04.834929 15120 raft_consensus.cc:482] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:23:04.834957 15120 raft_consensus.cc:3037] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:04.835667 15120 raft_consensus.cc:504] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.835822 15120 leader_election.cc:288] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [CANDIDATE]: Term 1 election: Requested vote from peers ce4ce81490bf41de8b8f6747be5e945a (127.5.148.1:41081), 8b4957cf85ba48baae33b8cd4b1bd35a (127.5.148.2:45623)
I20260812 14:23:04.835966 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d278eade1ec4b388b42fb657fa08865" candidate_uuid: "8caaa6377b6c4d1790191186b6715c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4ce81490bf41de8b8f6747be5e945a"
I20260812 14:23:04.836058 14716 raft_consensus.cc:3037] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:04.836041 14875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d278eade1ec4b388b42fb657fa08865" candidate_uuid: "8caaa6377b6c4d1790191186b6715c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a"
I20260812 14:23:04.836165 14875 raft_consensus.cc:3037] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:04.836819 14875 raft_consensus.cc:2454] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8caaa6377b6c4d1790191186b6715c90 in term 1.
I20260812 14:23:04.836906 14716 raft_consensus.cc:2454] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8caaa6377b6c4d1790191186b6715c90 in term 1.
I20260812 14:23:04.837075 14952 leader_election.cc:302] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8caaa6377b6c4d1790191186b6715c90, ce4ce81490bf41de8b8f6747be5e945a; no voters:
I20260812 14:23:04.837194 15120 raft_consensus.cc:2781] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:23:04.837352 15120 raft_consensus.cc:686] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [term 1 LEADER]: Becoming Leader. State: Replica: 8caaa6377b6c4d1790191186b6715c90, State: Running, Role: LEADER
I20260812 14:23:04.837459 15120 consensus_queue.cc:237] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.837870 15165 tablet_replica.cc:439] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90: TxnStatusTablet state changed. Reason: New leader 8caaa6377b6c4d1790191186b6715c90. Latest consensus state: current_term: 1 leader_uuid: "8caaa6377b6c4d1790191186b6715c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } }
I20260812 14:23:04.837983 15165 tablet_replica.cc:442] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:23:04.838181 15167 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 14:23:04.838315 15167 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 14:23:04.838215 14536 catalog_manager.cc:5495] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8caaa6377b6c4d1790191186b6715c90 (127.5.148.3). New cstate: current_term: 1 leader_uuid: "8caaa6377b6c4d1790191186b6715c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } health_report { overall_health: HEALTHY } } }
I20260812 14:23:04.848906 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.846050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:40879 (local address 127.0.0.1:54466)
0812 14:23:04.846124 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.846137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.846190 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.846384 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.846385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.846386 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:23:04.846441 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.846443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.847301 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.847301 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.847884 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.847886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.848403 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.848403 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.848460 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.848476 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:23:04.848765 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:23:04.848772 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.848804 (+ 32us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.848847 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 14:23:04.848938 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.846151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54466 (local address 127.5.148.3:40879)
0812 14:23:04.846214 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.846217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.846226 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:23:04.846268 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.846269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.846291 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.846382 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:23:04.846489 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.847226 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.847970 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.848318 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.848402 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.848571 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:23:04.848694 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:23:04.848768 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.848811 (+ 43us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.848831 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:23:04.850244 14716 raft_consensus.cc:1261] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Refusing update from remote peer 8caaa6377b6c4d1790191186b6715c90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:04.850354 14875 raft_consensus.cc:1261] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Refusing update from remote peer 8caaa6377b6c4d1790191186b6715c90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:04.850512 15165 consensus_queue.cc:1035] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:04.850625 15120 consensus_queue.cc:1035] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:04.851279 15125 tablet_replica.cc:439] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a: TxnStatusTablet state changed. Reason: New leader 8caaa6377b6c4d1790191186b6715c90. Latest consensus state: current_term: 1 leader_uuid: "8caaa6377b6c4d1790191186b6715c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } }
I20260812 14:23:04.851383 15125 tablet_replica.cc:442] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:23:04.851362 15122 tablet_replica.cc:439] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a: TxnStatusTablet state changed. Reason: New leader 8caaa6377b6c4d1790191186b6715c90. Latest consensus state: current_term: 1 leader_uuid: "8caaa6377b6c4d1790191186b6715c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } }
I20260812 14:23:04.851451 15122 tablet_replica.cc:442] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:23:04.852208 15120 tablet_replica.cc:439] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8caaa6377b6c4d1790191186b6715c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } }
I20260812 14:23:04.852296 15120 tablet_replica.cc:442] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:23:04.852380 15129 mvcc.cc:205] Tried to move back new op lower bound from 7317686619544342528 to 7317686619494895616. Current Snapshot: MvccSnapshot[applied={T|T < 7317686619544342528}]
I20260812 14:23:04.852500 15122 tablet_replica.cc:439] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8caaa6377b6c4d1790191186b6715c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } }
I20260812 14:23:04.852571 15122 tablet_replica.cc:442] T 7d278eade1ec4b388b42fb657fa08865 P ce4ce81490bf41de8b8f6747be5e945a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:23:04.852799 15120 tablet_replica.cc:439] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8caaa6377b6c4d1790191186b6715c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } }
I20260812 14:23:04.852908 15120 tablet_replica.cc:442] T 7d278eade1ec4b388b42fb657fa08865 P 8caaa6377b6c4d1790191186b6715c90: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:23:04.853484 15128 mvcc.cc:205] Tried to move back new op lower bound from 7317686619544342528 to 7317686619494895616. Current Snapshot: MvccSnapshot[applied={T|T < 7317686619544342528}]
I20260812 14:23:04.853462 15125 tablet_replica.cc:439] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8caaa6377b6c4d1790191186b6715c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } } }
I20260812 14:23:04.853817 15125 tablet_replica.cc:442] T 7d278eade1ec4b388b42fb657fa08865 P 8b4957cf85ba48baae33b8cd4b1bd35a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:23:04.857398 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.854559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53688 (local address 127.5.148.1:41081)
0812 14:23:04.854699 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.854701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.854708 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:23:04.854744 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.854745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.854763 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.854833 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:23:04.854960 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.855688 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.856552 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.856861 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.856935 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.857051 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:23:04.857189 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:23:04.857270 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.857294 (+ 24us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.857314 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260812 14:23:04.862847 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.860000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:39685 (local address 127.5.148.3:40879)
0812 14:23:04.860053 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.860056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.860224 (+ 168us) server_negotiation.cc:407] Connection header received
0812 14:23:04.860301 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.860302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.860311 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.860388 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:23:04.860574 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.861011 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.862130 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.862303 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.862354 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.862590 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:23:04.862622 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:23:04.862679 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.862764 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.862786 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:23:04.862923 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.859968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:40879 (local address 127.5.148.1:39685)
0812 14:23:04.860163 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.860175 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.860238 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.860415 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.860416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.860417 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:23:04.860506 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.860508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.861085 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.861086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.862045 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.862048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.862379 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.862380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.862456 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.862473 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:23:04.862697 (+ 224us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:23:04.862700 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.862756 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.862801 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260812 14:23:04.865679 15030 raft_consensus.cc:1261] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Refusing update from remote peer ce4ce81490bf41de8b8f6747be5e945a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:04.865712 14875 raft_consensus.cc:1261] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Refusing update from remote peer ce4ce81490bf41de8b8f6747be5e945a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:04.865962 15149 consensus_queue.cc:1035] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:04.866119 15122 consensus_queue.cc:1035] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:04.879745 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.872939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:59349 (local address 127.5.148.1:41081)
0812 14:23:04.872979 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.872981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.876455 (+ 3474us) server_negotiation.cc:407] Connection header received
0812 14:23:04.876525 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.876527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.876541 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.876605 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:23:04.876795 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.877306 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.878653 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.878986 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.879063 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.879382 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:23:04.879423 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:23:04.879506 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.879596 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.879644 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:23:04.879786 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.872879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:41081 (local address 127.5.148.3:59349)
0812 14:23:04.876362 (+ 3483us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:04.876378 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:04.876457 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:04.876612 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:04.876613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:04.876627 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:23:04.876722 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.876723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.877398 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.877399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.878517 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:04.878520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.879108 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.879109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:04.879194 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.879218 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:23:04.879509 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:23:04.879512 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:23:04.879586 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 14:23:04.879634 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3419,"thread_start_us":82,"threads_started":1}
I20260812 14:23:04.889293 14534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:23:04.889428 14534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:23:04.890069 14534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb: Sending DeleteTablet for 3 replicas of tablet da3e8def1c15412688510f77573ecade
I20260812 14:23:04.890336 14692 tablet_service.cc:1514] Processing DeleteTablet for tablet da3e8def1c15412688510f77573ecade with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:04 UTC) from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:53672
I20260812 14:23:04.890429 15010 tablet_service.cc:1514] Processing DeleteTablet for tablet da3e8def1c15412688510f77573ecade with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:04 UTC) from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:54456
I20260812 14:23:04.890457 15141 tablet_replica.cc:330] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a: stopping tablet replica
I20260812 14:23:04.890518 15142 tablet_replica.cc:330] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90: stopping tablet replica
I20260812 14:23:04.890532 15141 raft_consensus.cc:2229] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:23:04.890570 15142 raft_consensus.cc:2229] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:23:04.890617 15141 raft_consensus.cc:2258] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:04.890618 15142 raft_consensus.cc:2258] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:04.890832 14852 tablet_service.cc:1514] Processing DeleteTablet for tablet da3e8def1c15412688510f77573ecade with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:04 UTC) from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:57658
I20260812 14:23:04.890944 15143 tablet_replica.cc:330] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a: stopping tablet replica
I20260812 14:23:04.890995 15143 raft_consensus.cc:2229] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:23:04.890990 15142 ts_tablet_manager.cc:1905] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:04.891033 15143 raft_consensus.cc:2258] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:04.891749 15143 ts_tablet_manager.cc:1905] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:04.893126 15142 ts_tablet_manager.cc:1918] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:23:04.893213 15142 log.cc:1196] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/da3e8def1c15412688510f77573ecade
I20260812 14:23:04.893504 15142 ts_tablet_manager.cc:1939] T da3e8def1c15412688510f77573ecade P 8caaa6377b6c4d1790191186b6715c90: Deleting consensus metadata
I20260812 14:23:04.893661 15141 ts_tablet_manager.cc:1905] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:04.895370 15141 ts_tablet_manager.cc:1918] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:23:04.895431 15141 log.cc:1196] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/wal/wals/da3e8def1c15412688510f77573ecade
I20260812 14:23:04.895654 15141 ts_tablet_manager.cc:1939] T da3e8def1c15412688510f77573ecade P ce4ce81490bf41de8b8f6747be5e945a: Deleting consensus metadata
I20260812 14:23:04.895877 15143 ts_tablet_manager.cc:1918] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:23:04.895932 15143 log.cc:1196] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/da3e8def1c15412688510f77573ecade
I20260812 14:23:04.896200 15143 ts_tablet_manager.cc:1939] T da3e8def1c15412688510f77573ecade P 8b4957cf85ba48baae33b8cd4b1bd35a: Deleting consensus metadata
I20260812 14:23:04.896394 14515 catalog_manager.cc:4841] TS 8caaa6377b6c4d1790191186b6715c90 (127.5.148.3:40879): tablet da3e8def1c15412688510f77573ecade (table test-table [id=c33731eefa4547dcbb4561674c4a4508]) successfully deleted
I20260812 14:23:04.896545 14515 catalog_manager.cc:4841] TS 8b4957cf85ba48baae33b8cd4b1bd35a (127.5.148.2:45623): tablet da3e8def1c15412688510f77573ecade (table test-table [id=c33731eefa4547dcbb4561674c4a4508]) successfully deleted
I20260812 14:23:04.896838 14514 catalog_manager.cc:4841] TS ce4ce81490bf41de8b8f6747be5e945a (127.5.148.1:41081): tablet da3e8def1c15412688510f77573ecade (table test-table [id=c33731eefa4547dcbb4561674c4a4508]) successfully deleted
Aug 12 14:23:04 dist-test-slave-8w9v krb5kdc[14478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 14:23:04 dist-test-slave-8w9v krb5kdc[14478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:04 dist-test-slave-8w9v krb5kdc[14478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 14:23:04.922206 14594 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:04.916508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.5.148.62:38971)
0812 14:23:04.916564 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.916566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.916723 (+ 157us) server_negotiation.cc:407] Connection header received
0812 14:23:04.916820 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.916822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.916849 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.916919 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:04.917565 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.918075 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.918854 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.919066 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.919147 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.922154 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58478: BlockingRecv error: recv got EOF from 127.0.0.1:58478 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 12 14:23:04 dist-test-slave-8w9v krb5kdc[14478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM
I20260812 14:23:04.931046 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:04.923563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.5.148.62:38971)
0812 14:23:04.923606 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:04.923608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:04.923709 (+ 101us) server_negotiation.cc:407] Connection header received
0812 14:23:04.923792 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:04.923793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:04.923804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:04.923856 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:04.924437 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.924926 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.925628 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:04.925839 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:04.925931 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:04.928188 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:04.928190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:04.928190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:04.928199 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:04.929768 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:04.930158 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:04.930160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:04.930160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:04.930228 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:04.930431 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:04.930432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:04.930433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:04.930553 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:04.930671 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:04.930834 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 14:23:04.930930 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:23:04.932727 14534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:23:04.932929 14534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:23:04.934314 14692 tablet_service.cc:1467] Processing CreateTablet for tablet 360f78a4a2db480b9c8e4701af82e20b (DEFAULT_TABLE table=test-table [id=222391ae9c7d4a52b4f948c294a5aec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:04.934376 14852 tablet_service.cc:1467] Processing CreateTablet for tablet 360f78a4a2db480b9c8e4701af82e20b (DEFAULT_TABLE table=test-table [id=222391ae9c7d4a52b4f948c294a5aec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:04.934473 14692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 360f78a4a2db480b9c8e4701af82e20b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:04.934484 14852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 360f78a4a2db480b9c8e4701af82e20b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:04.935909 15119 tablet_bootstrap.cc:492] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a: Bootstrap starting.
I20260812 14:23:04.935909 15116 tablet_bootstrap.cc:492] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a: Bootstrap starting.
I20260812 14:23:04.936297 15010 tablet_service.cc:1467] Processing CreateTablet for tablet 360f78a4a2db480b9c8e4701af82e20b (DEFAULT_TABLE table=test-table [id=222391ae9c7d4a52b4f948c294a5aec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:04.936419 15116 tablet_bootstrap.cc:654] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:04.936431 15010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 360f78a4a2db480b9c8e4701af82e20b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:04.937350 15116 tablet_bootstrap.cc:492] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a: No bootstrap required, opened a new log
I20260812 14:23:04.937397 15116 ts_tablet_manager.cc:1397] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:04.937554 15116 raft_consensus.cc:348] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.937620 15116 raft_consensus.cc:374] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:04.937638 15116 raft_consensus.cc:729] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4ce81490bf41de8b8f6747be5e945a, State: Initialized, Role: FOLLOWER
I20260812 14:23:04.937685 15116 consensus_queue.cc:260] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.937781 15116 ts_tablet_manager.cc:1428] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:23:04.937918 15115 tablet_bootstrap.cc:492] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90: Bootstrap starting.
I20260812 14:23:04.938225 15115 tablet_bootstrap.cc:654] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:04.939016 15119 tablet_bootstrap.cc:654] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:04.940505 15115 tablet_bootstrap.cc:492] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90: No bootstrap required, opened a new log
I20260812 14:23:04.940559 15115 ts_tablet_manager.cc:1397] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 14:23:04.940706 15115 raft_consensus.cc:348] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.940819 15115 raft_consensus.cc:374] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:04.940843 15115 raft_consensus.cc:729] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8caaa6377b6c4d1790191186b6715c90, State: Initialized, Role: FOLLOWER
I20260812 14:23:04.940889 15115 consensus_queue.cc:260] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.940989 15115 ts_tablet_manager.cc:1428] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:04.942482 15119 tablet_bootstrap.cc:492] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a: No bootstrap required, opened a new log
I20260812 14:23:04.942559 15119 ts_tablet_manager.cc:1397] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260812 14:23:04.942694 15119 raft_consensus.cc:348] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.942786 15119 raft_consensus.cc:374] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:04.942808 15119 raft_consensus.cc:729] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4957cf85ba48baae33b8cd4b1bd35a, State: Initialized, Role: FOLLOWER
I20260812 14:23:04.942862 15119 consensus_queue.cc:260] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:04.942952 15119 ts_tablet_manager.cc:1428] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:05.067605 15175 raft_consensus.cc:482] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:23:05.067726 15175 raft_consensus.cc:504] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:05.067945 15175 leader_election.cc:288] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce4ce81490bf41de8b8f6747be5e945a (127.5.148.1:41081), 8b4957cf85ba48baae33b8cd4b1bd35a (127.5.148.2:45623)
I20260812 14:23:05.068223 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360f78a4a2db480b9c8e4701af82e20b" candidate_uuid: "8caaa6377b6c4d1790191186b6715c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4ce81490bf41de8b8f6747be5e945a" is_pre_election: true
I20260812 14:23:05.068248 14875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360f78a4a2db480b9c8e4701af82e20b" candidate_uuid: "8caaa6377b6c4d1790191186b6715c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" is_pre_election: true
I20260812 14:23:05.068337 14716 raft_consensus.cc:2454] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8caaa6377b6c4d1790191186b6715c90 in term 0.
I20260812 14:23:05.068353 14875 raft_consensus.cc:2454] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8caaa6377b6c4d1790191186b6715c90 in term 0.
I20260812 14:23:05.068538 14954 leader_election.cc:302] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b4957cf85ba48baae33b8cd4b1bd35a, 8caaa6377b6c4d1790191186b6715c90; no voters:
I20260812 14:23:05.068662 15175 raft_consensus.cc:2781] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:23:05.068734 15175 raft_consensus.cc:482] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:23:05.068766 15175 raft_consensus.cc:3037] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:05.069533 15175 raft_consensus.cc:504] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:05.069722 15175 leader_election.cc:288] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [CANDIDATE]: Term 1 election: Requested vote from peers ce4ce81490bf41de8b8f6747be5e945a (127.5.148.1:41081), 8b4957cf85ba48baae33b8cd4b1bd35a (127.5.148.2:45623)
I20260812 14:23:05.069892 14875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360f78a4a2db480b9c8e4701af82e20b" candidate_uuid: "8caaa6377b6c4d1790191186b6715c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a"
I20260812 14:23:05.069904 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360f78a4a2db480b9c8e4701af82e20b" candidate_uuid: "8caaa6377b6c4d1790191186b6715c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4ce81490bf41de8b8f6747be5e945a"
I20260812 14:23:05.069970 14716 raft_consensus.cc:3037] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:05.069989 14875 raft_consensus.cc:3037] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:05.070684 14875 raft_consensus.cc:2454] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8caaa6377b6c4d1790191186b6715c90 in term 1.
I20260812 14:23:05.070875 14716 raft_consensus.cc:2454] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8caaa6377b6c4d1790191186b6715c90 in term 1.
I20260812 14:23:05.070902 14954 leader_election.cc:302] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b4957cf85ba48baae33b8cd4b1bd35a, 8caaa6377b6c4d1790191186b6715c90; no voters:
I20260812 14:23:05.071025 15175 raft_consensus.cc:2781] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:23:05.071094 15175 raft_consensus.cc:686] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [term 1 LEADER]: Becoming Leader. State: Replica: 8caaa6377b6c4d1790191186b6715c90, State: Running, Role: LEADER
I20260812 14:23:05.071146 15175 consensus_queue.cc:237] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:05.071781 14534 catalog_manager.cc:5495] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8caaa6377b6c4d1790191186b6715c90 (127.5.148.3). New cstate: current_term: 1 leader_uuid: "8caaa6377b6c4d1790191186b6715c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } health_report { overall_health: HEALTHY } } }
I20260812 14:23:05.088443 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:05.085994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.5.148.3:40879)
0812 14:23:05.086049 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:05.086052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:05.086063 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:23:05.086149 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:05.086151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:05.086167 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:05.086230 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:23:05.086402 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:05.086874 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:05.087683 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:05.087877 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:05.087971 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:05.088155 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:23:05.088197 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:23:05.088260 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:05.088325 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 14:23:05.088349 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:23:05.089457 14875 raft_consensus.cc:1261] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Refusing update from remote peer 8caaa6377b6c4d1790191186b6715c90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:05.089661 14716 raft_consensus.cc:1261] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Refusing update from remote peer 8caaa6377b6c4d1790191186b6715c90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:05.089720 15165 consensus_queue.cc:1035] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:05.089881 15175 consensus_queue.cc:1035] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:05.093138 14534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:23:05.093230 14534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:23:05.093914 14534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb: Sending DeleteTablet for 3 replicas of tablet 360f78a4a2db480b9c8e4701af82e20b
I20260812 14:23:05.094178 14692 tablet_service.cc:1514] Processing DeleteTablet for tablet 360f78a4a2db480b9c8e4701af82e20b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:05 UTC) from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:53672
I20260812 14:23:05.094300 15141 tablet_replica.cc:330] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a: stopping tablet replica
I20260812 14:23:05.094306 14852 tablet_service.cc:1514] Processing DeleteTablet for tablet 360f78a4a2db480b9c8e4701af82e20b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:05 UTC) from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:57658
I20260812 14:23:05.094384 15141 raft_consensus.cc:2229] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:23:05.094434 15141 raft_consensus.cc:2258] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:05.094467 15010 tablet_service.cc:1514] Processing DeleteTablet for tablet 360f78a4a2db480b9c8e4701af82e20b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:05 UTC) from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:54456
I20260812 14:23:05.094517 15143 tablet_replica.cc:330] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a: stopping tablet replica
I20260812 14:23:05.094475 14534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 14:23:05.094558 15142 tablet_replica.cc:330] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90: stopping tablet replica
I20260812 14:23:05.094591 15143 raft_consensus.cc:2229] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:23:05.094610 15142 raft_consensus.cc:2229] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [term 1 LEADER]: Raft consensus shutting down.
W20260812 14:23:05.094619 14534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:23:05.094642 15143 raft_consensus.cc:2258] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:05.094705 15142 raft_consensus.cc:2258] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:05.095017 15141 ts_tablet_manager.cc:1905] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:05.095070 15142 ts_tablet_manager.cc:1905] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:05.096422 15143 ts_tablet_manager.cc:1905] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:05.096629 15142 ts_tablet_manager.cc:1918] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:23:05.096661 15141 ts_tablet_manager.cc:1918] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:23:05.096690 15142 log.cc:1196] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/360f78a4a2db480b9c8e4701af82e20b
I20260812 14:23:05.096714 15141 log.cc:1196] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-0/wal/wals/360f78a4a2db480b9c8e4701af82e20b
I20260812 14:23:05.096903 15142 ts_tablet_manager.cc:1939] T 360f78a4a2db480b9c8e4701af82e20b P 8caaa6377b6c4d1790191186b6715c90: Deleting consensus metadata
I20260812 14:23:05.096961 15141 ts_tablet_manager.cc:1939] T 360f78a4a2db480b9c8e4701af82e20b P ce4ce81490bf41de8b8f6747be5e945a: Deleting consensus metadata
I20260812 14:23:05.097226 14515 catalog_manager.cc:4841] TS 8caaa6377b6c4d1790191186b6715c90 (127.5.148.3:40879): tablet 360f78a4a2db480b9c8e4701af82e20b (table test-table [id=222391ae9c7d4a52b4f948c294a5aec9]) successfully deleted
I20260812 14:23:05.097311 14514 catalog_manager.cc:4841] TS ce4ce81490bf41de8b8f6747be5e945a (127.5.148.1:41081): tablet 360f78a4a2db480b9c8e4701af82e20b (table test-table [id=222391ae9c7d4a52b4f948c294a5aec9]) successfully deleted
I20260812 14:23:05.097915 15010 tablet_service.cc:1467] Processing CreateTablet for tablet c1d6120449b44b008275e3de6bc3ab74 (DEFAULT_TABLE table=test-table [id=a0fb314fcdd04ee1a58b1767c1cbe21f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:05.097945 14692 tablet_service.cc:1467] Processing CreateTablet for tablet c1d6120449b44b008275e3de6bc3ab74 (DEFAULT_TABLE table=test-table [id=a0fb314fcdd04ee1a58b1767c1cbe21f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:05.098037 15010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1d6120449b44b008275e3de6bc3ab74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:05.098037 14692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1d6120449b44b008275e3de6bc3ab74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:05.099375 15116 tablet_bootstrap.cc:492] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a: Bootstrap starting.
I20260812 14:23:05.099380 15115 tablet_bootstrap.cc:492] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90: Bootstrap starting.
I20260812 14:23:05.099409 14852 tablet_service.cc:1467] Processing CreateTablet for tablet c1d6120449b44b008275e3de6bc3ab74 (DEFAULT_TABLE table=test-table [id=a0fb314fcdd04ee1a58b1767c1cbe21f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:05.099725 15115 tablet_bootstrap.cc:654] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:05.099728 15116 tablet_bootstrap.cc:654] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:05.100225 15143 ts_tablet_manager.cc:1918] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:23:05.100294 15143 log.cc:1196] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/360f78a4a2db480b9c8e4701af82e20b
I20260812 14:23:05.100553 15143 ts_tablet_manager.cc:1939] T 360f78a4a2db480b9c8e4701af82e20b P 8b4957cf85ba48baae33b8cd4b1bd35a: Deleting consensus metadata
I20260812 14:23:05.100560 15115 tablet_bootstrap.cc:492] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90: No bootstrap required, opened a new log
I20260812 14:23:05.100611 15115 ts_tablet_manager.cc:1397] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:05.100713 14852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1d6120449b44b008275e3de6bc3ab74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:05.100739 15115 raft_consensus.cc:348] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:05.100832 15115 raft_consensus.cc:374] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:05.100865 15115 raft_consensus.cc:729] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8caaa6377b6c4d1790191186b6715c90, State: Initialized, Role: FOLLOWER
I20260812 14:23:05.100907 15115 consensus_queue.cc:260] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:05.101006 15115 ts_tablet_manager.cc:1428] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:23:05.102061 15119 tablet_bootstrap.cc:492] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a: Bootstrap starting.
I20260812 14:23:05.102288 15116 tablet_bootstrap.cc:492] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a: No bootstrap required, opened a new log
I20260812 14:23:05.102348 15116 ts_tablet_manager.cc:1397] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 14:23:05.102427 15119 tablet_bootstrap.cc:654] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:05.102492 15116 raft_consensus.cc:348] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:05.102582 15116 raft_consensus.cc:374] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:05.102610 15116 raft_consensus.cc:729] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4ce81490bf41de8b8f6747be5e945a, State: Initialized, Role: FOLLOWER
I20260812 14:23:05.102667 15116 consensus_queue.cc:260] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:05.102769 15116 ts_tablet_manager.cc:1428] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:23:05.103168 14515 catalog_manager.cc:4841] TS 8b4957cf85ba48baae33b8cd4b1bd35a (127.5.148.2:45623): tablet 360f78a4a2db480b9c8e4701af82e20b (table test-table [id=222391ae9c7d4a52b4f948c294a5aec9]) successfully deleted
I20260812 14:23:05.103314 15119 tablet_bootstrap.cc:492] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a: No bootstrap required, opened a new log
I20260812 14:23:05.103363 15119 ts_tablet_manager.cc:1397] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:23:05.103477 15119 raft_consensus.cc:348] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:05.103550 15119 raft_consensus.cc:374] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:05.103569 15119 raft_consensus.cc:729] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4957cf85ba48baae33b8cd4b1bd35a, State: Initialized, Role: FOLLOWER
I20260812 14:23:05.103613 15119 consensus_queue.cc:260] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:05.103720 15119 ts_tablet_manager.cc:1428] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:05.237200 15125 raft_consensus.cc:482] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:23:05.237318 15125 raft_consensus.cc:504] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:05.237582 15125 leader_election.cc:288] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce4ce81490bf41de8b8f6747be5e945a (127.5.148.1:41081), 8caaa6377b6c4d1790191186b6715c90 (127.5.148.3:40879)
I20260812 14:23:05.237864 15030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d6120449b44b008275e3de6bc3ab74" candidate_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8caaa6377b6c4d1790191186b6715c90" is_pre_election: true
I20260812 14:23:05.237891 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d6120449b44b008275e3de6bc3ab74" candidate_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4ce81490bf41de8b8f6747be5e945a" is_pre_election: true
I20260812 14:23:05.238005 15030 raft_consensus.cc:2454] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b4957cf85ba48baae33b8cd4b1bd35a in term 0.
I20260812 14:23:05.238001 14716 raft_consensus.cc:2454] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b4957cf85ba48baae33b8cd4b1bd35a in term 0.
I20260812 14:23:05.238220 14796 leader_election.cc:302] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b4957cf85ba48baae33b8cd4b1bd35a, ce4ce81490bf41de8b8f6747be5e945a; no voters:
I20260812 14:23:05.238348 15125 raft_consensus.cc:2781] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:23:05.238389 15125 raft_consensus.cc:482] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:23:05.238447 15125 raft_consensus.cc:3037] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:05.239229 15125 raft_consensus.cc:504] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:05.239398 15125 leader_election.cc:288] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [CANDIDATE]: Term 1 election: Requested vote from peers ce4ce81490bf41de8b8f6747be5e945a (127.5.148.1:41081), 8caaa6377b6c4d1790191186b6715c90 (127.5.148.3:40879)
I20260812 14:23:05.239578 15030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d6120449b44b008275e3de6bc3ab74" candidate_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8caaa6377b6c4d1790191186b6715c90"
I20260812 14:23:05.239578 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d6120449b44b008275e3de6bc3ab74" candidate_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4ce81490bf41de8b8f6747be5e945a"
I20260812 14:23:05.239667 15030 raft_consensus.cc:3037] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:05.239667 14716 raft_consensus.cc:3037] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:05.240540 15030 raft_consensus.cc:2454] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b4957cf85ba48baae33b8cd4b1bd35a in term 1.
I20260812 14:23:05.240540 14716 raft_consensus.cc:2454] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b4957cf85ba48baae33b8cd4b1bd35a in term 1.
I20260812 14:23:05.240782 14797 leader_election.cc:302] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b4957cf85ba48baae33b8cd4b1bd35a, 8caaa6377b6c4d1790191186b6715c90; no voters:
I20260812 14:23:05.240918 15125 raft_consensus.cc:2781] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:23:05.241042 15125 raft_consensus.cc:686] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 LEADER]: Becoming Leader. State: Replica: 8b4957cf85ba48baae33b8cd4b1bd35a, State: Running, Role: LEADER
I20260812 14:23:05.241119 15125 consensus_queue.cc:237] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } }
I20260812 14:23:05.241796 14533 catalog_manager.cc:5495] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b4957cf85ba48baae33b8cd4b1bd35a (127.5.148.2). New cstate: current_term: 1 leader_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b4957cf85ba48baae33b8cd4b1bd35a" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 45623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 } health_report { overall_health: UNKNOWN } } }
I20260812 14:23:05.251868 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:05.249284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.5.148.2:45623)
0812 14:23:05.249334 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:05.249336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:05.249346 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:23:05.249382 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:05.249382 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:05.249392 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:05.249442 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:23:05.249579 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:05.250108 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:05.251208 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:05.251432 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:05.251492 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:05.251618 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:23:05.251646 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:23:05.251686 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:05.251738 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 14:23:05.251760 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:23:05.257045 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:05.254444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:45157 (local address 127.5.148.3:40879)
0812 14:23:05.254518 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:05.254520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:05.254625 (+ 105us) server_negotiation.cc:407] Connection header received
0812 14:23:05.254666 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:05.254667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:05.254677 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:05.254726 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:23:05.254833 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:05.255308 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:05.256420 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:05.256608 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:05.256665 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:05.256829 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:23:05.256864 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:23:05.256903 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:05.256970 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 14:23:05.256986 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 14:23:05.257064 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:05.254248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:40879 (local address 127.5.148.2:45157)
0812 14:23:05.254561 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:05.254571 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:05.254624 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:05.254727 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:05.254727 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:05.254728 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:23:05.254784 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:05.254785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:05.255385 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:05.255386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:05.256297 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:05.256300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:05.256677 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:05.256678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:05.256736 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:05.256753 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:23:05.256916 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:23:05.256918 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:23:05.256957 (+ 39us) client_negotiation.cc:240] Negotiation successful
0812 14:23:05.256997 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":217,"threads_started":1}
I20260812 14:23:05.259414 15030 raft_consensus.cc:1261] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Refusing update from remote peer 8b4957cf85ba48baae33b8cd4b1bd35a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:05.259402 14716 raft_consensus.cc:1261] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Refusing update from remote peer 8b4957cf85ba48baae33b8cd4b1bd35a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:05.259639 15209 consensus_queue.cc:1035] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce4ce81490bf41de8b8f6747be5e945a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:05.259766 15125 consensus_queue.cc:1035] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8caaa6377b6c4d1790191186b6715c90" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:05.267618 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:05.265085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:45623 (local address 127.5.148.3:60823)
0812 14:23:05.265135 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:05.265148 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:05.265195 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:05.265319 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:05.265320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:05.265321 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:23:05.265374 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:05.265376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:05.266052 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:05.266053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:05.266680 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:05.266682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:05.267125 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:05.267126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:05.267191 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:05.267199 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:23:05.267436 (+ 237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:23:05.267439 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:23:05.267489 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:23:05.267548 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 14:23:05.267613 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:05.265142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:60823 (local address 127.5.148.2:45623)
0812 14:23:05.265183 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:05.265185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:05.265195 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:23:05.265250 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:05.265251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:05.265262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:05.265316 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:23:05.265450 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:05.265971 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:05.266770 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:05.267041 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:05.267113 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:05.267288 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:23:05.267334 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:23:05.267401 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:05.267492 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 14:23:05.267519 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:23:05.274164 14533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:23:05.274287 14533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:23:05.275004 14533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cd451a6a91140c2be387269249dbdcb: Sending DeleteTablet for 3 replicas of tablet c1d6120449b44b008275e3de6bc3ab74
I20260812 14:23:05.275288 14692 tablet_service.cc:1514] Processing DeleteTablet for tablet c1d6120449b44b008275e3de6bc3ab74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:05 UTC) from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:53672
I20260812 14:23:05.275300 15010 tablet_service.cc:1514] Processing DeleteTablet for tablet c1d6120449b44b008275e3de6bc3ab74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:05 UTC) from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:54456
I20260812 14:23:05.275326 14852 tablet_service.cc:1514] Processing DeleteTablet for tablet c1d6120449b44b008275e3de6bc3ab74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:05 UTC) from {username='oryx', principal='oryx/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:57658
I20260812 14:23:05.275398 15141 tablet_replica.cc:330] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a: stopping tablet replica
I20260812 14:23:05.275424 15143 tablet_replica.cc:330] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a: stopping tablet replica
I20260812 14:23:05.275452 15141 raft_consensus.cc:2229] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:23:05.275480 15143 raft_consensus.cc:2229] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:23:05.275491 15141 raft_consensus.cc:2258] T c1d6120449b44b008275e3de6bc3ab74 P ce4ce81490bf41de8b8f6747be5e945a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:05.275532 15143 raft_consensus.cc:2258] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:05.275568 15142 tablet_replica.cc:330] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90: stopping tablet replica
I20260812 14:23:05.275635 15142 raft_consensus.cc:2229] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:23:05.275671 15142 raft_consensus.cc:2258] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:05.275821 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 14620
I20260812 14:23:05.276043 15142 ts_tablet_manager.cc:1905] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:05.277817 15142 ts_tablet_manager.cc:1918] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:23:05.277884 15142 log.cc:1196] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/c1d6120449b44b008275e3de6bc3ab74
I20260812 14:23:05.278067 15142 ts_tablet_manager.cc:1939] T c1d6120449b44b008275e3de6bc3ab74 P 8caaa6377b6c4d1790191186b6715c90: Deleting consensus metadata
I20260812 14:23:05.278371 14515 catalog_manager.cc:4841] TS 8caaa6377b6c4d1790191186b6715c90 (127.5.148.3:40879): tablet c1d6120449b44b008275e3de6bc3ab74 (table test-table [id=a0fb314fcdd04ee1a58b1767c1cbe21f]) successfully deleted
I20260812 14:23:05.278414 15143 ts_tablet_manager.cc:1905] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:05.280400 15143 ts_tablet_manager.cc:1918] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:23:05.280472 15143 log.cc:1196] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/c1d6120449b44b008275e3de6bc3ab74
I20260812 14:23:05.280702 15143 ts_tablet_manager.cc:1939] T c1d6120449b44b008275e3de6bc3ab74 P 8b4957cf85ba48baae33b8cd4b1bd35a: Deleting consensus metadata
I20260812 14:23:05.281158 14515 catalog_manager.cc:4841] TS 8b4957cf85ba48baae33b8cd4b1bd35a (127.5.148.2:45623): tablet c1d6120449b44b008275e3de6bc3ab74 (table test-table [id=a0fb314fcdd04ee1a58b1767c1cbe21f]) successfully deleted
W20260812 14:23:05.288360 14514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:41081 (error 108)
I20260812 14:23:05.288756 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 14780
I20260812 14:23:05.289036 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:05.288801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:41081 (local address 127.0.0.1:53694)
0812 14:23:05.288945 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:05.288986 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:41081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
W20260812 14:23:05.289121 14514 catalog_manager.cc:4584] TS ce4ce81490bf41de8b8f6747be5e945a (127.5.148.1:41081): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c1d6120449b44b008275e3de6bc3ab74: Network error: Client connection negotiation failed: client connection to 127.5.148.1:41081: connect: Connection refused (error 111)
I20260812 14:23:05.299834 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 14935
I20260812 14:23:05.309679 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 14494
2026-08-12T14:23:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3058 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:23:05 dist-test-slave-8w9v krb5kdc[15222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:23:05 dist-test-slave-8w9v krb5kdc[15222](info): set up 2 sockets
Aug 12 14:23:05 dist-test-slave-8w9v krb5kdc[15222](info): commencing operation
krb5kdc: starting...
W20260812 14:23:06.461652 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:23:06 dist-test-slave-8w9v krb5kdc[15222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:23:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:23:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:23:06.608448 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:35003
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:35003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:23:06.699806 15236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:06.700059 15236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:06.700088 15236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:06.701442 15236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:23:06.701494 15236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:06.701514 15236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:06.701530 15236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:23:06.701546 15236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:23:06.703186 15236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:35003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:35003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:06.703435 15236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:06.703645 15236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:06.709031 15246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:06.709031 15241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:06.709196 15244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:06.709445 15236 server_base.cc:1034] running on GCE node
I20260812 14:23:06.709719 15236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:06.710048 15236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:06.711192 15236 hybrid_clock.cc:648] HybridClock initialized: now 1786544586711170 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T14:23:06Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1402 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:23:06 dist-test-slave-8w9v krb5kdc[15251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:23:06 dist-test-slave-8w9v krb5kdc[15251](info): set up 2 sockets
Aug 12 14:23:06 dist-test-slave-8w9v krb5kdc[15251](info): commencing operation
krb5kdc: starting...
W20260812 14:23:07.849510 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:23:07 dist-test-slave-8w9v krb5kdc[15251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:23:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:23:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:23:07.987802 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:35095
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:35095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:23:08.075065 15268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:08.075304 15268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:08.075330 15268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:08.076645 15268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:23:08.076697 15268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:08.076713 15268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:08.076726 15268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:23:08.076740 15268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:23:08.078408 15268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:35095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:35095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:08.078723 15268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:08.079015 15268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:08.084679 15274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:08.084661 15277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:08.084712 15275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:08.084910 15268 server_base.cc:1034] running on GCE node
I20260812 14:23:08.085224 15268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:08.085592 15268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:08.086777 15268 hybrid_clock.cc:648] HybridClock initialized: now 1786544588086768 us; error 52 us; skew 500 ppm
Aug 12 14:23:08 dist-test-slave-8w9v krb5kdc[15251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544588, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:08.089687 15268 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:23:08.090233 15268 webserver.cc:460] Webserver started at http://127.5.148.62:41845/ using document root <none> and password file <none>
I20260812 14:23:08.090514 15268 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:08.090590 15268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:08.090749 15268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:08.091924 15268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "5021b34a5637f9d2fd170e52f867a4b0"
server_key_iv: "6acbb93ffc92ef46e017d68d0a0fc28d"
server_key_version: "encryptionkey@0"
I20260812 14:23:08.092334 15268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "5021b34a5637f9d2fd170e52f867a4b0"
server_key_iv: "6acbb93ffc92ef46e017d68d0a0fc28d"
server_key_version: "encryptionkey@0"
I20260812 14:23:08.093787 15268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:23:08.094672 15287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:08.094899 15268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:23:08.095023 15268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "5021b34a5637f9d2fd170e52f867a4b0"
server_key_iv: "6acbb93ffc92ef46e017d68d0a0fc28d"
server_key_version: "encryptionkey@0"
I20260812 14:23:08.095111 15268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:08.103802 15268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:08.105893 15268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:08.106078 15268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:08.110462 15268 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:35095
I20260812 14:23:08.110498 15343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:35095 every 8 connection(s)
I20260812 14:23:08.110893 15268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:23:08.111891 15344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:08.114135 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 15268
I20260812 14:23:08.114262 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:23:08.114468 5712 external_mini_cluster.cc:1428] Setting key 7a0b99607c1dd3f8d73d2478d24d8e9a
I20260812 14:23:08.114741 15344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437: Bootstrap starting.
I20260812 14:23:08.115504 15344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:08.115823 15344 log.cc:824] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437: Log is configured to *not* fsync() on all Append() calls
I20260812 14:23:08.116756 15344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437: No bootstrap required, opened a new log
I20260812 14:23:08.118923 15344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35095 } }
I20260812 14:23:08.119138 15344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:08.119169 15344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d6bdf22f2a4d6e8b5242bf7eda5437, State: Initialized, Role: FOLLOWER
I20260812 14:23:08.119326 15344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35095 } }
I20260812 14:23:08.119410 15344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:23:08.119453 15344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:23:08.119503 15344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:08.120339 15344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35095 } }
I20260812 14:23:08.120491 15344 leader_election.cc:302] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35d6bdf22f2a4d6e8b5242bf7eda5437; no voters:
I20260812 14:23:08.120697 15344 leader_election.cc:288] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:23:08.120788 15349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:23:08.120997 15349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 1 LEADER]: Becoming Leader. State: Replica: 35d6bdf22f2a4d6e8b5242bf7eda5437, State: Running, Role: LEADER
I20260812 14:23:08.121176 15349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35095 } }
I20260812 14:23:08.121013 15344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 14:23:08 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:23:08.122969 15350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35095 } } }
I20260812 14:23:08.123171 15351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35d6bdf22f2a4d6e8b5242bf7eda5437. Latest consensus state: current_term: 1 leader_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35095 } } }
I20260812 14:23:08.123286 15350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [sys.catalog]: This master's current role is: LEADER
I20260812 14:23:08.123293 15351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [sys.catalog]: This master's current role is: LEADER
W20260812 14:23:08.123433 15366 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:23:08.123486 15366 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:23:08.123588 15368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:23:08.123766 15368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:23:08.125474 15368 catalog_manager.cc:1347] Generated new cluster ID: 326f8396742a4075b7d7aee7a8c94824
I20260812 14:23:08.125532 15368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:23:08.127451 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:08.115083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.5.148.62:35095)
0812 14:23:08.115282 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:08.115292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:08.115329 (+ 37us) server_negotiation.cc:407] Connection header received
0812 14:23:08.115830 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:08.115837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:08.115922 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:08.116094 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:08.116957 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:08.117906 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:08.119081 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:08.119728 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:08.119833 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:08.122992 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:08.122997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:08.122998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:08.123012 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:08.125290 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:08.125751 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:08.125754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:08.125754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:08.125864 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:08.126082 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:08.126083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:08.126083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:08.126422 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:08.126566 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:08.127060 (+ 494us) server_negotiation.cc:299] Negotiation successful
0812 14:23:08.127211 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
I20260812 14:23:08.160156 15368 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:23:08.160778 15368 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:23:08.164557 15368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437: Generated new TSK 0
I20260812 14:23:08.164788 15368 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:23:08.204661 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35095
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:23:08.292927 15372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:08.293206 15372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:08.293251 15372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:08.294523 15372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:08.294600 15372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:08.294644 15372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:23:08.296218 15372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:08.296509 15372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:08.296758 15372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:08.302480 15380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:08.302690 15372 server_base.cc:1034] running on GCE node
W20260812 14:23:08.302537 15383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:08.302482 15381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:08.302976 15372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:08.303225 15372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:08.304396 15372 hybrid_clock.cc:648] HybridClock initialized: now 1786544588304380 us; error 30 us; skew 500 ppm
Aug 12 14:23:08 dist-test-slave-8w9v krb5kdc[15251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544588, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:08.307044 15372 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:23:08.307560 15372 webserver.cc:460] Webserver started at http://127.5.148.1:36081/ using document root <none> and password file <none>
I20260812 14:23:08.307765 15372 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:08.307806 15372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:08.307901 15372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:08.309054 15372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "4c02a44459c54d5f81c7e22daa9e995a"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "8c449c60bfaa720f76ed277344149ceb"
server_key_iv: "24bf4285b31c1344018bdfa9bb93d942"
server_key_version: "encryptionkey@0"
I20260812 14:23:08.309369 15372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "4c02a44459c54d5f81c7e22daa9e995a"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "8c449c60bfaa720f76ed277344149ceb"
server_key_iv: "24bf4285b31c1344018bdfa9bb93d942"
server_key_version: "encryptionkey@0"
I20260812 14:23:08.310748 15372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:23:08.311636 15391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:08.311807 15372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:23:08.311903 15372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "4c02a44459c54d5f81c7e22daa9e995a"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "8c449c60bfaa720f76ed277344149ceb"
server_key_iv: "24bf4285b31c1344018bdfa9bb93d942"
server_key_version: "encryptionkey@0"
I20260812 14:23:08.311967 15372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:08.323894 15372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:08.326371 15372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:08.326489 15372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:08.326697 15372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:08.326982 15372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:08.327015 15372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:08.327037 15372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:08.327050 15372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:08.333240 15372 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:44087
I20260812 14:23:08.333307 15526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:44087 every 8 connection(s)
I20260812 14:23:08.333714 15372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:23:08 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544588, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:23:08.341477 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 15372
I20260812 14:23:08.341609 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:23:08.341809 5712 external_mini_cluster.cc:1428] Setting key a66eb64a958058255cc70d596e3eb6c1
I20260812 14:23:08.343725 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:08.334526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:53859)
0812 14:23:08.334740 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:08.334787 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:08.335302 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:08.335665 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:08.335669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:08.336342 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:08.336558 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:08.336563 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:08.337264 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:08.337264 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:08.338099 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:08.338102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:08.338513 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:08.338514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:08.338629 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:08.339132 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:08.339141 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:08.340618 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:08.342284 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:08.342292 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:08.342294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:08.342599 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:08.342826 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:08.342827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:08.342827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:08.342941 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:08.343224 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:08.343229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:08.343350 (+ 121us) client_negotiation.cc:769] Sending connection context
0812 14:23:08.343436 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 14:23:08.343558 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":127,"threads_started":1}
I20260812 14:23:08.343722 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:08.334572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:53859 (local address 127.5.148.62:35095)
0812 14:23:08.334639 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:08.334641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:08.334887 (+ 246us) server_negotiation.cc:407] Connection header received
0812 14:23:08.335424 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:08.335425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:08.335437 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:08.335507 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:08.336662 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:08.337177 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:08.338179 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:08.338398 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:08.338455 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:08.340707 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:08.340708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:08.340709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:08.340715 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:08.342209 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:08.342695 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:08.342696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:08.342696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:08.342764 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:08.343005 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:08.343006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:08.343006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:08.343156 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:08.343226 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:08.343482 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 14:23:08.343583 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:23:08.344360 15527 heartbeater.cc:347] Connected to a master server at 127.5.148.62:35095
I20260812 14:23:08.344480 15527 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:08.344699 15527 heartbeater.cc:511] Master 127.5.148.62:35095 requested a full tablet report, sending...
I20260812 14:23:08.345361 15307 ts_manager.cc:194] Registered new tserver with Master: 4c02a44459c54d5f81c7e22daa9e995a (127.5.148.1:44087)
I20260812 14:23:08.346174 15307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:53859
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:23:08.371981 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35095
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:23:08.458889 15532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:08.459138 15532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:08.459167 15532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:08.460464 15532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:08.460515 15532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:08.460543 15532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:23:08.462139 15532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:08.462443 15532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:08.462721 15532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:08.468220 15542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:08.468230 15540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:08.468241 15539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:08.468521 15532 server_base.cc:1034] running on GCE node
I20260812 14:23:08.468793 15532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:08.469070 15532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:08.470229 15532 hybrid_clock.cc:648] HybridClock initialized: now 1786544588470217 us; error 31 us; skew 500 ppm
Aug 12 14:23:08 dist-test-slave-8w9v krb5kdc[15251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544588, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:08.473192 15532 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:23:08.473694 15532 webserver.cc:460] Webserver started at http://127.5.148.2:35409/ using document root <none> and password file <none>
I20260812 14:23:08.473950 15532 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:08.474025 15532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:08.474187 15532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:08.475296 15532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "5bbd8eb396e340dd88186c68070d63fa"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "9344772332728b6f1936082bbe56f538"
server_key_iv: "2e70c5e78719789edf71ce5d69666ee2"
server_key_version: "encryptionkey@0"
I20260812 14:23:08.475687 15532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "5bbd8eb396e340dd88186c68070d63fa"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "9344772332728b6f1936082bbe56f538"
server_key_iv: "2e70c5e78719789edf71ce5d69666ee2"
server_key_version: "encryptionkey@0"
I20260812 14:23:08.477205 15532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:23:08.478026 15551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:08.478202 15532 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:23:08.478319 15532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "5bbd8eb396e340dd88186c68070d63fa"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "9344772332728b6f1936082bbe56f538"
server_key_iv: "2e70c5e78719789edf71ce5d69666ee2"
server_key_version: "encryptionkey@0"
I20260812 14:23:08.478399 15532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:08.492422 15532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:08.494710 15532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:08.494877 15532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:08.495142 15532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:08.495507 15532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:08.495550 15532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:08.495596 15532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:08.495622 15532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:08.502017 15532 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:43685
I20260812 14:23:08.502104 15680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:43685 every 8 connection(s)
I20260812 14:23:08.502471 15532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:23:08.508332 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 15532
I20260812 14:23:08.508448 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:23:08.508672 5712 external_mini_cluster.cc:1428] Setting key b96e5d091858a145331c2201947cdf12
Aug 12 14:23:08 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544588, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:23:08.513129 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:08.503296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:58745 (local address 127.5.148.62:35095)
0812 14:23:08.503386 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:08.503388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:08.503593 (+ 205us) server_negotiation.cc:407] Connection header received
0812 14:23:08.504115 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:08.504116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:08.504169 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:08.504235 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:08.505239 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:08.505752 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:08.506769 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:08.506953 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:08.507005 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:08.509653 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:08.509655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:08.509656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:08.509666 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:08.511374 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:08.511975 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:08.511976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:08.511977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:08.512037 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:08.512402 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:08.512403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:08.512403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:08.512535 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:08.512593 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:08.512985 (+ 392us) server_negotiation.cc:299] Negotiation successful
0812 14:23:08.513061 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 14:23:08.513484 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:08.503249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:58745)
0812 14:23:08.503453 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:08.503497 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:08.504025 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:08.504296 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:08.504300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:08.504925 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:08.505148 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:08.505153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:08.505850 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:08.505851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:08.506684 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:08.506687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:08.507044 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:08.507045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:08.507157 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:08.507702 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:08.507711 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:08.509547 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:08.511509 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:08.511520 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:08.511522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:08.511887 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:08.512112 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:08.512113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:08.512113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:08.512328 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:08.512609 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:08.512618 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:08.512816 (+ 198us) client_negotiation.cc:769] Sending connection context
0812 14:23:08.512965 (+ 149us) client_negotiation.cc:240] Negotiation successful
0812 14:23:08.513187 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":108,"threads_started":1}
I20260812 14:23:08.514084 15681 heartbeater.cc:347] Connected to a master server at 127.5.148.62:35095
I20260812 14:23:08.514204 15681 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:08.514420 15681 heartbeater.cc:511] Master 127.5.148.62:35095 requested a full tablet report, sending...
I20260812 14:23:08.514899 15307 ts_manager.cc:194] Registered new tserver with Master: 5bbd8eb396e340dd88186c68070d63fa (127.5.148.2:43685)
I20260812 14:23:08.515321 15307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:58745
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:23:08.540552 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35095
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:23:08.629209 15686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:08.629454 15686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:08.629482 15686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:08.630689 15686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:08.630733 15686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:08.630759 15686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:23:08.632355 15686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:08.632586 15686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:08.632791 15686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:08.638278 15691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:08.638350 15692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:08.638398 15694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:08.638736 15686 server_base.cc:1034] running on GCE node
I20260812 14:23:08.638943 15686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:08.639246 15686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:08.640424 15686 hybrid_clock.cc:648] HybridClock initialized: now 1786544588640407 us; error 31 us; skew 500 ppm
Aug 12 14:23:08 dist-test-slave-8w9v krb5kdc[15251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544588, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:08.643249 15686 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:23:08.643796 15686 webserver.cc:460] Webserver started at http://127.5.148.3:33809/ using document root <none> and password file <none>
I20260812 14:23:08.644054 15686 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:08.644150 15686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:08.644310 15686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:08.645848 15686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "932dbe3299e547f0953784c6001b1f29"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "aa09f08771e7548b6b3b473d7e61f07c"
server_key_iv: "2443816638a05863e78f02312aef5490"
server_key_version: "encryptionkey@0"
I20260812 14:23:08.646239 15686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "932dbe3299e547f0953784c6001b1f29"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "aa09f08771e7548b6b3b473d7e61f07c"
server_key_iv: "2443816638a05863e78f02312aef5490"
server_key_version: "encryptionkey@0"
I20260812 14:23:08.647686 15686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:23:08.648561 15701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:08.648737 15686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:23:08.648846 15686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "932dbe3299e547f0953784c6001b1f29"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "aa09f08771e7548b6b3b473d7e61f07c"
server_key_iv: "2443816638a05863e78f02312aef5490"
server_key_version: "encryptionkey@0"
I20260812 14:23:08.648923 15686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:08.657202 15686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:08.659673 15686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:08.659857 15686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:08.660176 15686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:08.660523 15686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:08.660581 15686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:08.660624 15686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:08.660653 15686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:08.667387 15686 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:45571
I20260812 14:23:08.667443 15830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:45571 every 8 connection(s)
I20260812 14:23:08.667785 15686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:23:08 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544588, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:23:08.677047 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:08.668589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:48235 (local address 127.5.148.62:35095)
0812 14:23:08.668674 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:08.668676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:08.668826 (+ 150us) server_negotiation.cc:407] Connection header received
0812 14:23:08.669314 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:08.669315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:08.669326 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:08.669377 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:08.670351 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:08.670860 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:08.671885 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:08.672074 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:08.672184 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:08.674512 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:08.674514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:08.674514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:08.674521 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:08.675740 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:08.676176 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:08.676178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:08.676178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:08.676239 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:08.676477 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:08.676478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:08.676478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:08.676587 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:08.676643 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:08.676886 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 14:23:08.676955 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 14:23:08.677207 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:08.668523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:48235)
0812 14:23:08.668694 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:08.668734 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:08.669227 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:08.669443 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:08.669447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:08.670043 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:08.670253 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:08.670257 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:08.670957 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:08.670958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:08.671804 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:08.671806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:08.672244 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:08.672245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:08.672383 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:08.672981 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:08.672990 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:08.674423 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:08.675816 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:08.675823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:08.675825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:08.676059 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:08.676302 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:08.676303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:08.676303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:08.676416 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:08.676646 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:08.676651 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:08.676774 (+ 123us) client_negotiation.cc:769] Sending connection context
0812 14:23:08.676862 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 14:23:08.676990 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260812 14:23:08.677304 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 15686
I20260812 14:23:08.677408 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:23:08.677639 5712 external_mini_cluster.cc:1428] Setting key 8023daad5bcd7ea141116d17544bda56
I20260812 14:23:08.677768 15831 heartbeater.cc:347] Connected to a master server at 127.5.148.62:35095
I20260812 14:23:08.677868 15831 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:08.678100 15831 heartbeater.cc:511] Master 127.5.148.62:35095 requested a full tablet report, sending...
I20260812 14:23:08.678617 15306 ts_manager.cc:194] Registered new tserver with Master: 932dbe3299e547f0953784c6001b1f29 (127.5.148.3:45571)
I20260812 14:23:08.679018 15306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:48235
I20260812 14:23:08.689818 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.1.keytab.
Entry for principal oryx/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.1.keytab.
WARNING: no policy specified for oryx/127.5.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.2.keytab.
Entry for principal oryx/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.2.keytab.
WARNING: no policy specified for oryx/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.3.keytab.
Entry for principal oryx/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.3.keytab.
I20260812 14:23:08.779989 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 15372
I20260812 14:23:08.789781 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 15532
I20260812 14:23:08.798164 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 15686
I20260812 14:23:08.808419 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 15268
I20260812 14:23:08.817286 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:35095
--webserver_interface=127.5.148.62
--webserver_port=41845
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:35095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:23:08.903282 15840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:08.903509 15840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:08.903534 15840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:08.904862 15840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:23:08.904914 15840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:08.904929 15840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:08.904942 15840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:23:08.904994 15840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:23:08.906526 15840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:35095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:35095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=41845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:08.906798 15840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:08.907049 15840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:08.912487 15848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:08.912493 15845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:08.912532 15840 server_base.cc:1034] running on GCE node
W20260812 14:23:08.912493 15846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:08.912953 15840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:08.913267 15840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:08.914460 15840 hybrid_clock.cc:648] HybridClock initialized: now 1786544588914464 us; error 46 us; skew 500 ppm
Aug 12 14:23:08 dist-test-slave-8w9v krb5kdc[15251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544588, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:08.917212 15840 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:23:08.917821 15840 webserver.cc:460] Webserver started at http://127.5.148.62:41845/ using document root <none> and password file <none>
I20260812 14:23:08.918098 15840 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:08.918208 15840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:08.919854 15840 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:23:08.920686 15857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:08.920867 15840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:23:08.920984 15840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "5021b34a5637f9d2fd170e52f867a4b0"
server_key_iv: "6acbb93ffc92ef46e017d68d0a0fc28d"
server_key_version: "encryptionkey@0"
I20260812 14:23:08.921397 15840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:08.941344 15840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:08.943799 15840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:08.944000 15840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:08.948596 15840 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:35095
I20260812 14:23:08.948630 15917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:35095 every 8 connection(s)
I20260812 14:23:08.949044 15840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:23:08.950562 15918 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:23:08.951201 15918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437: Bootstrap starting.
I20260812 14:23:08.953406 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 15840
I20260812 14:23:08.953774 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:44087
--local_ip_for_outbound_sockets=127.5.148.1
--tserver_master_addrs=127.5.148.62:35095
--webserver_port=36081
--webserver_interface=127.5.148.1
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.148.1
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:23:08.961596 15918 log.cc:824] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437: Log is configured to *not* fsync() on all Append() calls
I20260812 14:23:08.966952 15918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:23:08.967360 15918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437: Bootstrap complete.
I20260812 14:23:08.970847 15918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35095 } }
I20260812 14:23:08.971184 15918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d6bdf22f2a4d6e8b5242bf7eda5437, State: Initialized, Role: FOLLOWER
I20260812 14:23:08.971364 15918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35095 } }
I20260812 14:23:08.971434 15918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:23:08.971488 15918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:23:08.971536 15918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:23:08.973001 15918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35095 } }
I20260812 14:23:08.973125 15918 leader_election.cc:302] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35d6bdf22f2a4d6e8b5242bf7eda5437; no voters:
I20260812 14:23:08.973335 15918 leader_election.cc:288] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 14:23:08.973425 15922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 14:23:08.973594 15918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:23:08.973642 15922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [term 2 LEADER]: Becoming Leader. State: Replica: 35d6bdf22f2a4d6e8b5242bf7eda5437, State: Running, Role: LEADER
I20260812 14:23:08.973882 15922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35095 } }
I20260812 14:23:08.974220 15922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35d6bdf22f2a4d6e8b5242bf7eda5437. Latest consensus state: current_term: 2 leader_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35095 } } }
I20260812 14:23:08.974220 15924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d6bdf22f2a4d6e8b5242bf7eda5437" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35095 } } }
I20260812 14:23:08.974372 15924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [sys.catalog]: This master's current role is: LEADER
I20260812 14:23:08.974540 15922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437 [sys.catalog]: This master's current role is: LEADER
I20260812 14:23:08.974645 15932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:23:08.975276 15932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:23:08.975512 15932 catalog_manager.cc:1259] Loaded cluster ID: 326f8396742a4075b7d7aee7a8c94824
I20260812 14:23:08.975701 15932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:23:08.976713 15932 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:23:08.976926 15932 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 35d6bdf22f2a4d6e8b5242bf7eda5437: Loaded TSK: 0
I20260812 14:23:08.977154 15932 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:23:09.055557 15920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:09.055800 15920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:09.055841 15920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:09.057088 15920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:09.057163 15920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:09.057209 15920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:23:09.058704 15920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.1.keytab
--principal=oryx/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:44087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=36081
--tserver_master_addrs=127.5.148.62:35095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:09.058988 15920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:09.059228 15920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:09.064519 15950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:09.064540 15946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:09.064662 15948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:09.064751 15920 server_base.cc:1034] running on GCE node
I20260812 14:23:09.065039 15920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:09.065366 15920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:09.066567 15920 hybrid_clock.cc:648] HybridClock initialized: now 1786544589066546 us; error 33 us; skew 500 ppm
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544589, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:09.069773 15920 init.cc:377] Logged in from keytab as oryx/127.5.148.1@KRBTEST.COM (short username oryx)
I20260812 14:23:09.070277 15920 webserver.cc:460] Webserver started at http://127.5.148.1:36081/ using document root <none> and password file <none>
I20260812 14:23:09.070562 15920 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:09.070659 15920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:09.072238 15920 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:23:09.072971 15957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:09.073158 15920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:09.073253 15920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "4c02a44459c54d5f81c7e22daa9e995a"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "8c449c60bfaa720f76ed277344149ceb"
server_key_iv: "24bf4285b31c1344018bdfa9bb93d942"
server_key_version: "encryptionkey@0"
I20260812 14:23:09.073683 15920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:09.096213 15920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:09.098822 15920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:09.099015 15920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:09.099313 15920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:09.099689 15920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:09.099745 15920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:09.099782 15920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:09.099809 15920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:09.106204 15920 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:44087
I20260812 14:23:09.106256 16087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:44087 every 8 connection(s)
I20260812 14:23:09.106619 15920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:23:09.111539 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 15920
I20260812 14:23:09.111845 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:43685
--local_ip_for_outbound_sockets=127.5.148.2
--tserver_master_addrs=127.5.148.62:35095
--webserver_port=35409
--webserver_interface=127.5.148.2
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.148.2
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:09.117497 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:09.107436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:47285)
0812 14:23:09.107622 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:09.107668 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:09.108248 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:09.108759 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:09.108763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:09.109354 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:09.109564 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.109569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.111265 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.111266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.112630 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.112634 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.113560 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.113561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.113699 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.114346 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:09.114356 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:09.117230 (+ 2874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
W20260812 14:23:09.117897 16089 heartbeater.cc:644] Failed to heartbeat to 127.5.148.62:35095 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.148.62:35095: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
W20260812 14:23:09.117548 16092 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:09.107541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:47285 (local address 127.5.148.62:35095)
0812 14:23:09.107759 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:09.107768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:09.107817 (+ 49us) server_negotiation.cc:407] Connection header received
0812 14:23:09.108495 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:09.108503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:09.108558 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:09.108714 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:09.109696 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.111157 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.112728 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.113348 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.113457 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.117385 (+ 3928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:47285: BlockingRecv error: recv got EOF from 127.5.148.1:47285 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":98,"threads_started":1}
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:09.123685 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:09.118134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:37409)
0812 14:23:09.118194 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:09.118207 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:09.118261 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:09.118574 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:09.118576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:09.118749 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:09.118814 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.118816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.119879 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.119880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.120594 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.120596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.121118 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.121118 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.121168 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.121665 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:09.121669 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:09.123609 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 14:23:09.123917 16094 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:09.118238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:37409 (local address 127.5.148.62:35095)
0812 14:23:09.118407 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:09.118409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:09.118427 (+ 18us) server_negotiation.cc:407] Connection header received
0812 14:23:09.118454 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:09.118455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:09.118469 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:09.118893 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:09.118932 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.119774 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.120679 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.121031 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.121115 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.123829 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:37409: BlockingRecv error: recv got EOF from 127.5.148.1:37409 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:09.128901 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:09.124156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:47793)
0812 14:23:09.124200 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:09.124212 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:09.124261 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:09.124411 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:09.124412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:09.124584 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:09.124627 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.124629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.125382 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.125383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.126248 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.126251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.126581 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.126582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.126637 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.127045 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:09.127050 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:09.128831 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22}
W20260812 14:23:09.128995 16094 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:09.124206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:47793 (local address 127.5.148.62:35095)
0812 14:23:09.124257 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:09.124259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:09.124267 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:23:09.124328 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:09.124329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:09.124337 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:09.124384 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:09.124707 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.125310 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.126336 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.126515 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.126565 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.128915 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:47793: BlockingRecv error: recv got EOF from 127.5.148.1:47793 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260812 14:23:09.129184 16089 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:23:09.202526 16093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:09.202751 16093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:09.202775 16093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:09.203981 16093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:09.204026 16093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:09.204053 16093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:23:09.205520 16093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.2.keytab
--principal=oryx/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:43685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=35409
--tserver_master_addrs=127.5.148.62:35095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:09.205770 16093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:09.206015 16093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:09.211181 16103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:09.211182 16101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:09.211445 16093 server_base.cc:1034] running on GCE node
W20260812 14:23:09.211524 16100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:09.211773 16093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:09.212046 16093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:09.213222 16093 hybrid_clock.cc:648] HybridClock initialized: now 1786544589213225 us; error 49 us; skew 500 ppm
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544589, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:09.215749 16093 init.cc:377] Logged in from keytab as oryx/127.5.148.2@KRBTEST.COM (short username oryx)
I20260812 14:23:09.216285 16093 webserver.cc:460] Webserver started at http://127.5.148.2:35409/ using document root <none> and password file <none>
I20260812 14:23:09.216522 16093 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:09.216606 16093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:09.217962 16093 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:09.218546 16113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:09.218722 16093 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:23:09.218823 16093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "5bbd8eb396e340dd88186c68070d63fa"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "9344772332728b6f1936082bbe56f538"
server_key_iv: "2e70c5e78719789edf71ce5d69666ee2"
server_key_version: "encryptionkey@0"
I20260812 14:23:09.219193 16093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:09.227696 16093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:09.229641 16093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:09.229809 16093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:09.230054 16093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:09.230374 16093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:09.230430 16093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:09.230472 16093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:09.230497 16093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:09.236653 16093 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:43685
I20260812 14:23:09.236696 16241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:43685 every 8 connection(s)
I20260812 14:23:09.237108 16093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:23:09.239476 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 16093
I20260812 14:23:09.239790 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:45571
--local_ip_for_outbound_sockets=127.5.148.3
--tserver_master_addrs=127.5.148.62:35095
--webserver_port=33809
--webserver_interface=127.5.148.3
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.148.3
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:09.245815 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:09.237929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:53221)
0812 14:23:09.238173 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:09.238220 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:09.238749 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:09.238986 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:09.238989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:09.239844 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:09.240174 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.240181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.240901 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.240903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.242033 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.242036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.242542 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.242543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.242649 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.243228 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:09.243236 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:09.245658 (+ 2422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":106,"threads_started":1}
W20260812 14:23:09.245899 16094 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:09.237963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:53221 (local address 127.5.148.62:35095)
0812 14:23:09.238043 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:09.238045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:09.238317 (+ 272us) server_negotiation.cc:407] Connection header received
0812 14:23:09.238850 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:09.238852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:09.238869 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:09.238926 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:09.240286 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.240791 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.242137 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.242448 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.242530 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.245797 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:53221: BlockingRecv error: recv got EOF from 127.5.148.2:53221 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 14:23:09.246163 16242 heartbeater.cc:644] Failed to heartbeat to 127.5.148.62:35095 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.148.62:35095: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:09.250844 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:09.246441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:48923)
0812 14:23:09.246484 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:09.246495 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:09.246560 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:09.246691 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:09.246692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:09.246831 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:09.246875 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.246876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.247523 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.247524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.248024 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.248025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.248446 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.248447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.248518 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.249040 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:09.249044 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:09.250757 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 14:23:09.250900 16094 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:09.246514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:48923 (local address 127.5.148.62:35095)
0812 14:23:09.246581 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:09.246583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:09.246592 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:23:09.246616 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:09.246617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:09.246629 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:09.246686 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:09.246960 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.247438 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.248111 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.248355 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.248430 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.250833 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:48923: BlockingRecv error: recv got EOF from 127.5.148.2:48923 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:09.255175 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:09.251143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:54123)
0812 14:23:09.251185 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:09.251195 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:09.251235 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:09.251416 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:09.251417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:09.251549 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:09.251575 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.251576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.252216 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.252217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.252701 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.252702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.253049 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.253050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.253102 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.253532 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:09.253536 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:09.255130 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 14:23:09.255296 16094 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:09.251241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:54123 (local address 127.5.148.62:35095)
0812 14:23:09.251303 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:09.251305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:09.251319 (+ 14us) server_negotiation.cc:407] Connection header received
0812 14:23:09.251338 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:09.251339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:09.251347 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:09.251412 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:09.251651 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.252101 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.252785 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.252975 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.253051 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.255238 (+ 2187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:54123: BlockingRecv error: recv got EOF from 127.5.148.2:54123 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260812 14:23:09.255407 16242 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:23:09.327229 16246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:09.327489 16246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:09.327535 16246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:09.328824 16246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:09.328899 16246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:09.328945 16246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:23:09.330466 16246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/krb5kdc/oryx_127.5.148.3.keytab
--principal=oryx/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:45571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=33809
--tserver_master_addrs=127.5.148.62:35095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:09.330760 16246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:09.331019 16246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:09.336571 16256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:09.336598 16258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:09.336644 16254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:09.336789 16246 server_base.cc:1034] running on GCE node
I20260812 14:23:09.337119 16246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:09.337409 16246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:09.338567 16246 hybrid_clock.cc:648] HybridClock initialized: now 1786544589338555 us; error 29 us; skew 500 ppm
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544589, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:09.341284 16246 init.cc:377] Logged in from keytab as oryx/127.5.148.3@KRBTEST.COM (short username oryx)
I20260812 14:23:09.341771 16246 webserver.cc:460] Webserver started at http://127.5.148.3:33809/ using document root <none> and password file <none>
I20260812 14:23:09.342039 16246 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:09.342135 16246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:09.343578 16246 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:23:09.344267 16265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:09.344455 16246 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:23:09.344563 16246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "932dbe3299e547f0953784c6001b1f29"
format_stamp: "Formatted at 2026-08-12 14:23:08 on dist-test-slave-8w9v"
server_key: "aa09f08771e7548b6b3b473d7e61f07c"
server_key_iv: "2443816638a05863e78f02312aef5490"
server_key_version: "encryptionkey@0"
I20260812 14:23:09.344945 16246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:09.366194 16246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:09.368309 16246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:09.368502 16246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:09.368767 16246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:09.369100 16246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:09.369158 16246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:09.369201 16246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:09.369249 16246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:09.375771 16246 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:45571
I20260812 14:23:09.375864 16395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:45571 every 8 connection(s)
I20260812 14:23:09.376272 16246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:23:09.376796 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 16246
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 14:23:09.384955 16094 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:09.377535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:42481 (local address 127.5.148.62:35095)
0812 14:23:09.377594 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:09.377596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:09.377610 (+ 14us) server_negotiation.cc:407] Connection header received
0812 14:23:09.378225 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:09.378226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:09.378247 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:09.378307 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:09.379317 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.379798 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.381171 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.381379 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.381473 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.384891 (+ 3418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:42481: BlockingRecv error: recv got EOF from 127.5.148.3:42481 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:23:09.385028 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:09.377191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:42481)
0812 14:23:09.377401 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:09.377455 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:09.378109 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:09.378382 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:09.378386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:09.378995 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:09.379200 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.379205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.379910 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.379911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.381054 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.381057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.381488 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.381489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.381629 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.382187 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:09.382198 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:09.384807 (+ 2609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
I20260812 14:23:09.385146 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:09.377698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.5.148.62:35095)
0812 14:23:09.377767 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:09.377769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:09.377777 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:23:09.377803 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:09.377804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:09.377817 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:09.377923 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:09.378573 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.379095 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.379888 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.380104 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.380208 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.381591 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:09.381595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:09.381595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:09.381611 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:09.383575 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:09.383972 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:09.383978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:09.383979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:09.384128 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:09.384367 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:09.384368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:09.384369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:09.384616 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:09.384777 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:09.384896 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 14:23:09.385028 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260812 14:23:09.385298 16396 heartbeater.cc:644] Failed to heartbeat to 127.5.148.62:35095 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.148.62:35095: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:09.390192 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:09.385576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:57257)
0812 14:23:09.385688 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:09.385700 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:09.385777 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:09.385952 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:09.385953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:09.386101 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:09.386174 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.386177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.387036 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.387037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.387592 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.387594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.387928 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.387929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.387988 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.388495 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:09.388500 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:09.390138 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260812 14:23:09.390300 16092 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:09.385634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:57257 (local address 127.5.148.62:35095)
0812 14:23:09.385671 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:09.385672 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:09.385754 (+ 82us) server_negotiation.cc:407] Connection header received
0812 14:23:09.385844 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:09.385844 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:09.385852 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:09.385946 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:09.386256 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.386949 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.387667 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.387848 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.387896 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.390230 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:57257: BlockingRecv error: recv got EOF from 127.5.148.3:57257 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:09 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:09.394604 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:09.390514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:57919)
0812 14:23:09.390584 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:09.390596 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:09.390644 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:09.390797 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:09.390798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:09.390926 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:09.390954 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.390955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.391566 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.391568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.392063 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:09.392064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.392447 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.392448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:09.392503 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.392890 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:09.392894 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:09.394514 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 14:23:09.394654 16092 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:09.390551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:57919 (local address 127.5.148.62:35095)
0812 14:23:09.390604 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:09.390606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:09.390628 (+ 22us) server_negotiation.cc:407] Connection header received
0812 14:23:09.390704 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:09.390705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:09.390713 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:09.390776 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:09.391031 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.391483 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.392161 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:09.392356 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:09.392446 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:09.394589 (+ 2143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:57919: BlockingRecv error: recv got EOF from 127.5.148.3:57919 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260812 14:23:09.394749 16396 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 14:23:10 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:10 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:10.134373 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:10.129622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:41993)
0812 14:23:10.129774 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:10.129791 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:10.129842 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:10.130000 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:10.130001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:10.130172 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:10.130225 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:10.130227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:10.130885 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:10.130886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:10.131434 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:10.131435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:10.131798 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:10.131799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:10.131841 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:10.132385 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:10.132390 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:10.134303 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
W20260812 14:23:10.134542 16401 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:10.129710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:41993 (local address 127.5.148.62:35095)
0812 14:23:10.129872 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:10.129874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:10.129882 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:23:10.129905 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:10.129906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:10.129916 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:10.129983 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:10.130299 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:10.130814 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:10.131518 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:10.131726 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:10.131785 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:10.134458 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:41993: BlockingRecv error: recv got EOF from 127.5.148.1:41993 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":105,"threads_started":1}
Aug 12 14:23:10 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:10 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:10.260566 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:10.255881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:43955)
0812 14:23:10.256024 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:10.256037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:10.256104 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:10.256378 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:10.256379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:10.256554 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:10.256614 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:10.256615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:10.257236 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:10.257236 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:10.257775 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:10.257776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:10.258108 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:10.258109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:10.258155 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:10.258608 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:10.258612 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:10.260493 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
W20260812 14:23:10.260663 16401 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:10.255927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:43955 (local address 127.5.148.62:35095)
0812 14:23:10.256008 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:10.256011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:10.256109 (+ 98us) server_negotiation.cc:407] Connection header received
0812 14:23:10.256232 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:10.256233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:10.256248 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:10.256348 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:10.256684 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:10.257166 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:10.257853 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:10.258038 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:10.258092 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:10.260594 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:43955: BlockingRecv error: recv got EOF from 127.5.148.2:43955 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 12 14:23:10 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:10 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:10.405193 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:10.395368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:35653)
0812 14:23:10.395549 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:10.395562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:10.395635 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:10.395793 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:10.395794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:10.395996 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:10.396048 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:10.396050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:10.396715 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:10.396716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:10.397270 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:10.397272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:10.397676 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:10.397677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:10.397718 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:10.403059 (+ 5341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:10.403065 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:10.405074 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":127,"threads_started":1}
W20260812 14:23:10.405241 16401 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:10.395453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:35653 (local address 127.5.148.62:35095)
0812 14:23:10.395558 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:10.395560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:10.395637 (+ 77us) server_negotiation.cc:407] Connection header received
0812 14:23:10.395710 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:10.395711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:10.395726 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:10.395774 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:10.396154 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:10.396645 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:10.397391 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:10.397590 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:10.397665 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:10.405170 (+ 7505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:35653: BlockingRecv error: recv got EOF from 127.5.148.3:35653 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 12 14:23:11 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:11 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:11.139880 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:11.134850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:51683)
0812 14:23:11.135081 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:11.135098 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:11.135175 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:11.135351 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:11.135352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:11.135581 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:11.135666 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:11.135668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:11.136499 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:11.136500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:11.137065 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:11.137067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:11.137393 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:11.137393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:11.137437 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:11.137891 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:11.137897 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:11.139774 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":105,"threads_started":1}
W20260812 14:23:11.139928 16405 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:11.134990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:51683 (local address 127.5.148.62:35095)
0812 14:23:11.135177 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:11.135180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:11.135195 (+ 15us) server_negotiation.cc:407] Connection header received
0812 14:23:11.135231 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:11.135232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:11.135244 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:11.135316 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:11.135749 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:11.136399 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:11.137138 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:11.137321 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:11.137371 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:11.139857 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:51683: BlockingRecv error: recv got EOF from 127.5.148.1:51683 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
Aug 12 14:23:11 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:11 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:11.265594 16406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:11.261102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:45341)
0812 14:23:11.261241 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:11.261257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:11.261312 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:11.261452 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:11.261453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:11.261628 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:11.261681 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:11.261683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:11.262334 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:11.262334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:11.262887 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:11.262889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:11.263231 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:11.263231 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:11.263272 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:11.263764 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:11.263769 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:11.265528 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
W20260812 14:23:11.265725 16405 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:11.261211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:45341 (local address 127.5.148.62:35095)
0812 14:23:11.261271 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:11.261273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:11.261299 (+ 26us) server_negotiation.cc:407] Connection header received
0812 14:23:11.261376 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:11.261377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:11.261388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:11.261435 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:11.261750 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:11.262263 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:11.262971 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:11.263158 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:11.263216 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:11.265625 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:45341: BlockingRecv error: recv got EOF from 127.5.148.2:45341 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 12 14:23:11 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:11 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:11.410874 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:11.405788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:40119)
0812 14:23:11.405932 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:11.405945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:11.406026 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:11.406213 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:11.406214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:11.406411 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:11.406481 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:11.406482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:11.407202 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:11.407204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:11.407757 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:11.407758 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:11.408180 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:11.408181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:11.408243 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:11.408769 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:11.408775 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:11.410813 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
W20260812 14:23:11.411024 16405 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:11.405989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:40119 (local address 127.5.148.62:35095)
0812 14:23:11.406070 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:11.406073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:11.406081 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:23:11.406110 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:11.406111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:11.406133 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:11.406204 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:11.406571 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:11.407090 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:11.407847 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:11.408051 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:11.408175 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:11.410942 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:40119: BlockingRecv error: recv got EOF from 127.5.148.3:40119 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 12 14:23:12 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:12 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:12.145555 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:12.140443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:36901)
0812 14:23:12.140622 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:12.140635 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:12.140706 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:12.140848 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:12.140849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:12.141030 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:12.141085 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:12.141086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:12.141788 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:12.141790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:12.142383 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:12.142385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:12.142850 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:12.142852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:12.142907 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:12.143503 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:12.143508 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:12.145492 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
W20260812 14:23:12.145706 16409 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:12.140535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:36901 (local address 127.5.148.62:35095)
0812 14:23:12.140647 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:12.140650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:12.140705 (+ 55us) server_negotiation.cc:407] Connection header received
0812 14:23:12.140766 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:12.140767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:12.140778 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:12.140831 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:12.141152 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:12.141677 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:12.142507 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:12.142743 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:12.142825 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:12.145606 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:36901: BlockingRecv error: recv got EOF from 127.5.148.1:36901 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
Aug 12 14:23:12 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:12 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:12.271642 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:12.266257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:56201)
0812 14:23:12.266475 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:12.266487 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:12.266589 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:12.266758 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:12.266760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:12.266980 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:12.267036 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:12.267038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:12.268040 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:12.268041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:12.268683 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:12.268685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:12.269077 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:12.269078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:12.269141 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:12.269704 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:12.269710 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:12.271548 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":112,"threads_started":1}
W20260812 14:23:12.271723 16409 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:12.266363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:56201 (local address 127.5.148.62:35095)
0812 14:23:12.266444 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:12.266447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:12.266567 (+ 120us) server_negotiation.cc:407] Connection header received
0812 14:23:12.266654 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:12.266655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:12.266669 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:12.266750 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:12.267135 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:12.267933 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:12.268777 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:12.268997 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:12.269077 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:12.271657 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:56201: BlockingRecv error: recv got EOF from 127.5.148.2:56201 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 12 14:23:12 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:12 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:12.416299 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:12.411421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:52741)
0812 14:23:12.411608 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:12.411619 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:12.411684 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:12.411817 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:12.411818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:12.411993 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:12.412049 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:12.412050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:12.412845 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:12.412846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:12.413397 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:12.413398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:12.413733 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:12.413734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:12.413775 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:12.414229 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:12.414233 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:12.416205 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":94,"threads_started":1}
W20260812 14:23:12.416457 16409 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:12.411530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:52741 (local address 127.5.148.62:35095)
0812 14:23:12.411575 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:12.411577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:12.411687 (+ 110us) server_negotiation.cc:407] Connection header received
0812 14:23:12.411736 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:12.411736 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:12.411751 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:12.411793 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:12.412163 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:12.412760 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:12.413486 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:12.413666 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:12.413718 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:12.416370 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:52741: BlockingRecv error: recv got EOF from 127.5.148.3:52741 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 12 14:23:13 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:13 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:13.151219 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:13.146096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:53897)
0812 14:23:13.146286 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:13.146297 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:13.146371 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:13.146520 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:13.146521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:13.146714 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:13.146769 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:13.146771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:13.147503 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:13.147504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:13.148330 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:13.148332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:13.148759 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:13.148760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:13.148819 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:13.149330 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:13.149336 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:13.151158 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":126,"threads_started":1}
W20260812 14:23:13.151351 16414 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:13.146179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:53897 (local address 127.5.148.62:35095)
0812 14:23:13.146303 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:13.146305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:13.146364 (+ 59us) server_negotiation.cc:407] Connection header received
0812 14:23:13.146430 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:13.146431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:13.146442 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:13.146506 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:13.146835 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:13.147417 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:13.148449 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:13.148665 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:13.148758 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:13.151260 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:53897: BlockingRecv error: recv got EOF from 127.5.148.1:53897 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
Aug 12 14:23:13 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:13 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:13.277797 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:13.272210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:42975)
0812 14:23:13.272381 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:13.272392 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:13.272473 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:13.272617 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:13.272617 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:13.272793 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:13.272845 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:13.272847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:13.273635 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:13.273637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:13.274330 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:13.274333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:13.274770 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:13.274771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:13.274836 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:13.275393 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:13.275399 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:13.277703 (+ 2304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":110,"threads_started":1}
W20260812 14:23:13.277920 16414 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:13.272298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:42975 (local address 127.5.148.62:35095)
0812 14:23:13.272374 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:13.272376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:13.272467 (+ 91us) server_negotiation.cc:407] Connection header received
0812 14:23:13.272541 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:13.272542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:13.272554 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:13.272600 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:13.272940 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:13.273505 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:13.274424 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:13.274682 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:13.274765 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:13.277843 (+ 3078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:42975: BlockingRecv error: recv got EOF from 127.5.148.2:42975 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 12 14:23:13 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:13 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:13.421702 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:13.416815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:36099)
0812 14:23:13.416967 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:13.416979 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:13.417055 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:13.417189 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:13.417190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:13.417407 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:13.417464 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:13.417465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:13.418191 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:13.418192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:13.418740 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:13.418741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:13.419095 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:13.419096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:13.419146 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:13.419742 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:13.419747 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:13.421625 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
W20260812 14:23:13.421857 16414 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:13.416915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:36099 (local address 127.5.148.62:35095)
0812 14:23:13.416973 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:13.416975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:13.417045 (+ 70us) server_negotiation.cc:407] Connection header received
0812 14:23:13.417103 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:13.417104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:13.417116 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:13.417172 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:13.417560 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:13.418081 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:13.418827 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:13.419015 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:13.419077 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:13.421782 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:36099: BlockingRecv error: recv got EOF from 127.5.148.3:36099 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 12 14:23:14 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:14 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:14.156526 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:14.151767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:41459)
0812 14:23:14.151909 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:14.151920 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:14.151988 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:14.152115 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:14.152116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:14.152334 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:14.152393 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:14.152395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:14.153080 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:14.153081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:14.153627 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:14.153629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:14.153979 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:14.153980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:14.154026 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:14.154510 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:14.154515 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:14.156453 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
W20260812 14:23:14.156682 16418 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:14.151808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:41459 (local address 127.5.148.62:35095)
0812 14:23:14.151935 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:14.151941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:14.151977 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:23:14.152034 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:14.152034 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:14.152044 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:14.152103 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:14.152459 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:14.152977 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:14.153705 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:14.153907 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:14.153962 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:14.156570 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:41459: BlockingRecv error: recv got EOF from 127.5.148.1:41459 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":50,"threads_started":1}
Aug 12 14:23:14 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:14 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:14.283427 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:14.278426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:41685)
0812 14:23:14.278611 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:14.278627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:14.278704 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:14.278858 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:14.278859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:14.279038 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:14.279090 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:14.279092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:14.279802 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:14.279803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:14.280437 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:14.280440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:14.280824 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:14.280825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:14.280872 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:14.281383 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:14.281389 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:14.283287 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":79,"threads_started":1}
W20260812 14:23:14.283541 16418 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:14.278589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:41685 (local address 127.5.148.62:35095)
0812 14:23:14.278689 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:14.278691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:14.278703 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:23:14.278761 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:14.278762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:14.278776 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:14.278850 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:14.279184 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:14.279724 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:14.280537 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:14.280745 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:14.280806 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:14.283451 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:41685: BlockingRecv error: recv got EOF from 127.5.148.2:41685 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 12 14:23:14 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:14 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:14.427561 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:14.422279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:53901)
0812 14:23:14.422444 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:14.422456 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:14.422552 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:14.422698 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:14.422698 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:14.422919 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:14.422974 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:14.422976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:14.423811 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:14.423813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:14.424409 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:14.424411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:14.424826 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:14.424827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:14.424885 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:14.425379 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:14.425384 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:14.427461 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":120,"threads_started":1}
W20260812 14:23:14.427628 16418 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:14.422460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:53901 (local address 127.5.148.62:35095)
0812 14:23:14.422530 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:14.422533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:14.422546 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:23:14.422624 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:14.422626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:14.422639 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:14.422694 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:14.423067 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:14.423711 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:14.424498 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:14.424732 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:14.424809 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:14.427560 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:53901: BlockingRecv error: recv got EOF from 127.5.148.3:53901 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 12 14:23:15 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:15 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:15.161657 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:15.157097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:33685)
0812 14:23:15.157249 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:15.157259 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:15.157312 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:15.157481 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:15.157481 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:15.157655 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:15.157708 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:15.157709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:15.158400 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:15.158402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:15.158978 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:15.158980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:15.159339 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:15.159340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:15.159398 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:15.159847 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:15.159852 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:15.161597 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
W20260812 14:23:15.161782 16423 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:15.157193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:33685 (local address 127.5.148.62:35095)
0812 14:23:15.157342 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:15.157348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:15.157358 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:23:15.157384 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:15.157385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:15.157395 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:15.157466 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:15.157773 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:15.158285 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:15.159064 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:15.159258 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:15.159312 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:15.161697 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:33685: BlockingRecv error: recv got EOF from 127.5.148.1:33685 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
Aug 12 14:23:15 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:15 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:15.288885 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:15.284096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:42733)
0812 14:23:15.284315 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:15.284327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:15.284423 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:15.284536 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:15.284536 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:15.284710 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:15.284762 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:15.284764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:15.285407 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:15.285407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:15.285960 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:15.285962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:15.286304 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:15.286305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:15.286352 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:15.286841 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:15.286847 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:15.288794 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":108,"threads_started":1}
W20260812 14:23:15.289044 16423 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:15.284169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:42733 (local address 127.5.148.62:35095)
0812 14:23:15.284224 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:15.284226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:15.284438 (+ 212us) server_negotiation.cc:407] Connection header received
0812 14:23:15.284467 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:15.284467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:15.284481 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:15.284523 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:15.284818 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:15.285314 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:15.286033 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:15.286238 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:15.286292 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:15.288982 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:42733: BlockingRecv error: recv got EOF from 127.5.148.2:42733 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 12 14:23:15 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:15 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:15.437523 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:15.428223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:43315)
0812 14:23:15.428403 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:15.428417 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:15.428486 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:15.428648 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:15.428649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:15.428847 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:15.428904 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:15.428906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:15.429565 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:15.429567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:15.430154 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:15.430156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:15.430527 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:15.430528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:15.430588 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:15.435156 (+ 4568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:15.435162 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:15.437402 (+ 2240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
W20260812 14:23:15.437618 16423 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:15.428362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:43315 (local address 127.5.148.62:35095)
0812 14:23:15.428459 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:15.428461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:15.428470 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:23:15.428557 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:15.428558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:15.428570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:15.428621 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:15.428970 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:15.429475 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:15.430231 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:15.430432 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:15.430484 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:15.437537 (+ 7053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:43315: BlockingRecv error: recv got EOF from 127.5.148.3:43315 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 14:23:16 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:16 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:16.166720 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:16.162206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:60107)
0812 14:23:16.162372 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:16.162390 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:16.162473 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:16.162605 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:16.162606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:16.162778 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:16.162831 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:16.162833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:16.163467 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:16.163468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:16.163981 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:16.163982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:16.164334 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:16.164334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:16.164377 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:16.164918 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:16.164924 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:16.166661 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
W20260812 14:23:16.166873 16429 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:16.162262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:60107 (local address 127.5.148.62:35095)
0812 14:23:16.162411 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:16.162413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:16.162467 (+ 54us) server_negotiation.cc:407] Connection header received
0812 14:23:16.162520 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:16.162521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:16.162532 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:16.162591 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:16.162889 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:16.163397 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:16.164057 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:16.164270 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:16.164323 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:16.166765 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:60107: BlockingRecv error: recv got EOF from 127.5.148.1:60107 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
Aug 12 14:23:16 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:16 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:16.294497 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:16.289480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:48049)
0812 14:23:16.289703 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:16.289716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:16.289822 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:16.290008 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:16.290009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:16.290204 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:16.290256 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:16.290257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:16.290977 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:16.290978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:16.291558 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:16.291559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:16.291962 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:16.291963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:16.292022 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:16.292582 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:16.292588 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:16.294395 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
W20260812 14:23:16.294551 16429 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:16.289576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:48049 (local address 127.5.148.62:35095)
0812 14:23:16.289687 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:16.289689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:16.289797 (+ 108us) server_negotiation.cc:407] Connection header received
0812 14:23:16.289912 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:16.289913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:16.289926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:16.290000 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:16.290374 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:16.290881 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:16.291633 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:16.291886 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:16.291962 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:16.294485 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:48049: BlockingRecv error: recv got EOF from 127.5.148.2:48049 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 12 14:23:16 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:16 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:16.448531 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:16.438260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:45655)
0812 14:23:16.438478 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:16.438491 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:16.438571 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:16.438727 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:16.438728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:16.438939 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:16.438991 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:16.438992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:16.439704 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:16.439706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:16.440287 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:16.440288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:16.440656 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:16.440657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:16.440720 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:16.446366 (+ 5646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:16.446373 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:16.448433 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":122,"threads_started":1}
W20260812 14:23:16.448598 16429 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:16.438375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:45655 (local address 127.5.148.62:35095)
0812 14:23:16.438450 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:16.438453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:16.438548 (+ 95us) server_negotiation.cc:407] Connection header received
0812 14:23:16.438631 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:16.438632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:16.438648 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:16.438719 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:16.439078 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:16.439613 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:16.440388 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:16.440581 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:16.440657 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:16.448533 (+ 7876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:45655: BlockingRecv error: recv got EOF from 127.5.148.3:45655 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 12 14:23:17 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:17 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:17.172335 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:17.167458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:35089)
0812 14:23:17.167626 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:17.167636 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:17.167684 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:17.167895 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:17.167896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:17.168072 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:17.168149 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:17.168151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:17.168795 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:17.168796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:17.169447 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:17.169451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:17.169947 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:17.169948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:17.169995 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:17.170443 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:17.170448 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:17.172274 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
W20260812 14:23:17.172473 16434 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:17.167589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:35089 (local address 127.5.148.62:35095)
0812 14:23:17.167735 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:17.167737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:17.167748 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:23:17.167781 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:17.167782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:17.167792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:17.167864 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:17.168219 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:17.168723 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:17.169534 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:17.169872 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:17.169944 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:17.172381 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:35089: BlockingRecv error: recv got EOF from 127.5.148.1:35089 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
Aug 12 14:23:17 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:17 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:17.300215 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:17.295081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:49761)
0812 14:23:17.295224 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:17.295241 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:17.295309 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:17.295467 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:17.295468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:17.295645 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:17.295699 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:17.295701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:17.296541 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:17.296542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:17.297086 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:17.297088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:17.297425 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:17.297427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:17.297478 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:17.298007 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:17.298012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:17.300088 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
W20260812 14:23:17.300302 16434 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:17.295141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:49761 (local address 127.5.148.62:35095)
0812 14:23:17.295217 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:17.295218 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:17.295308 (+ 90us) server_negotiation.cc:407] Connection header received
0812 14:23:17.295378 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:17.295379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:17.295392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:17.295446 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:17.295760 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:17.296449 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:17.297164 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:17.297353 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:17.297404 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:17.300239 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:49761: BlockingRecv error: recv got EOF from 127.5.148.2:49761 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 12 14:23:17 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:17 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:17.454023 16436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:17.449063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:58733)
0812 14:23:17.449302 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:17.449315 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:17.449412 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:17.449591 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:17.449592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:17.449787 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:17.449850 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:17.449851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:17.450553 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:17.450554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:17.451121 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:17.451123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:17.451496 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:17.451497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:17.451564 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:17.452086 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:17.452091 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:17.453921 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":142,"threads_started":1}
W20260812 14:23:17.454082 16434 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:17.449161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:58733 (local address 127.5.148.62:35095)
0812 14:23:17.449257 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:17.449260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:17.449390 (+ 130us) server_negotiation.cc:407] Connection header received
0812 14:23:17.449499 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:17.449500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:17.449519 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:17.449581 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:17.449937 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:17.450462 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:17.451216 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:17.451420 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:17.451497 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:17.454015 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:58733: BlockingRecv error: recv got EOF from 127.5.148.3:58733 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 12 14:23:18 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:18 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:18.177641 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:18.172815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:52111)
0812 14:23:18.172958 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:18.172967 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:18.173012 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:18.173168 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:18.173169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:18.173341 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:18.173391 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:18.173392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:18.174127 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:18.174128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:18.174675 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:18.174677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:18.175077 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:18.175079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:18.175137 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:18.175655 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:18.175660 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:18.177580 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
W20260812 14:23:18.177750 16438 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:18.172895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:52111 (local address 127.5.148.62:35095)
0812 14:23:18.173024 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:18.173026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:18.173058 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:23:18.173083 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:18.173084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:18.173095 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:18.173149 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:18.173455 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:18.174049 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:18.174750 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:18.174995 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:18.175049 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:18.177671 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:52111: BlockingRecv error: recv got EOF from 127.5.148.1:52111 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
Aug 12 14:23:18 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:18 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:18.305668 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:18.300751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:59249)
0812 14:23:18.300981 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:18.300993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:18.301063 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:18.301200 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:18.301201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:18.301377 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:18.301434 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:18.301436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:18.302089 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:18.302090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:18.302661 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:18.302664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:18.303023 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:18.303024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:18.303078 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:18.303619 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:18.303623 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:18.305607 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":125,"threads_started":1}
W20260812 14:23:18.305815 16438 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:18.300813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:59249 (local address 127.5.148.62:35095)
0812 14:23:18.300891 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:18.300893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:18.301078 (+ 185us) server_negotiation.cc:407] Connection header received
0812 14:23:18.301113 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:18.301114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:18.301127 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:18.301183 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:18.301500 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:18.301988 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:18.302745 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:18.302938 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:18.302990 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:18.305699 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:59249: BlockingRecv error: recv got EOF from 127.5.148.2:59249 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 12 14:23:18 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:18 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:18.459479 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:18.454605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:44415)
0812 14:23:18.454771 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:18.454782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:18.454853 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:18.455007 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:18.455008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:18.455204 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:18.455263 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:18.455265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:18.455918 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:18.455919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:18.456576 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:18.456579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:18.456919 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:18.456920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:18.456964 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:18.457452 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:18.457457 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:18.459418 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
W20260812 14:23:18.459643 16438 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:18.454677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:44415 (local address 127.5.148.62:35095)
0812 14:23:18.454748 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:18.454750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:18.454847 (+ 97us) server_negotiation.cc:407] Connection header received
0812 14:23:18.454903 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:18.454904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:18.454927 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:18.454983 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:18.455331 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:18.455826 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:18.456664 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:18.456850 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:18.456905 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:18.459546 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:44415: BlockingRecv error: recv got EOF from 127.5.148.3:44415 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 12 14:23:19 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:19 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:19.183327 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:19.178236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:42325)
0812 14:23:19.178415 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:19.178430 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:19.178506 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:19.178668 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:19.178669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:19.178867 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:19.178921 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:19.178923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:19.179727 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:19.179728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:19.180295 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:19.180297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:19.180674 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:19.180675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:19.180740 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:19.181243 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:19.181248 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:19.183216 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
W20260812 14:23:19.183387 16444 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:19.178309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:42325 (local address 127.5.148.62:35095)
0812 14:23:19.178471 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:19.178473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:19.178489 (+ 16us) server_negotiation.cc:407] Connection header received
0812 14:23:19.178572 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:19.178573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:19.178586 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:19.178652 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:19.179010 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:19.179638 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:19.180384 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:19.180587 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:19.180671 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:19.183317 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:42325: BlockingRecv error: recv got EOF from 127.5.148.1:42325 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
Aug 12 14:23:19 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:19 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:19.311298 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:19.306283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:42847)
0812 14:23:19.306457 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:19.306467 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:19.306558 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:19.306722 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:19.306723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:19.306922 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:19.306972 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:19.306974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:19.307671 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:19.307672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:19.308238 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:19.308240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:19.308608 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:19.308609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:19.308675 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:19.309161 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:19.309165 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:19.311188 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
W20260812 14:23:19.311340 16444 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:19.306453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:42847 (local address 127.5.148.62:35095)
0812 14:23:19.306535 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:19.306537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:19.306549 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:23:19.306629 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:19.306630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:19.306647 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:19.306714 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:19.307063 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:19.307570 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:19.308326 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:19.308529 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:19.308610 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:19.311274 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:42847: BlockingRecv error: recv got EOF from 127.5.148.2:42847 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 12 14:23:19 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:19 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:19.465288 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:19.460193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:54957)
0812 14:23:19.460387 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:19.460400 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:19.460458 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:19.460637 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:19.460639 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:19.460831 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:19.460888 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:19.460890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:19.461658 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:19.461659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:19.462191 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:19.462192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:19.462547 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:19.462548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:19.462589 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:19.463049 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:19.463053 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:19.465185 (+ 2132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
W20260812 14:23:19.465341 16444 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:19.460374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:54957 (local address 127.5.148.62:35095)
0812 14:23:19.460445 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:19.460447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:19.460503 (+ 56us) server_negotiation.cc:407] Connection header received
0812 14:23:19.460537 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:19.460538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:19.460552 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:19.460600 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:19.460971 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:19.461581 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:19.462274 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:19.462478 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:19.462537 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:19.465274 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:54957: BlockingRecv error: recv got EOF from 127.5.148.3:54957 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 12 14:23:20 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:20 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:20.188699 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:20.183935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:58171)
0812 14:23:20.184099 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:20.184111 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:20.184220 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:20.184356 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:20.184357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:20.184552 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:20.184605 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:20.184606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:20.185250 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:20.185251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:20.185788 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:20.185789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:20.186110 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:20.186111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:20.186154 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:20.186714 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:20.186718 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:20.188624 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":99,"threads_started":1}
W20260812 14:23:20.188858 16448 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:20.184044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:58171 (local address 127.5.148.62:35095)
0812 14:23:20.184227 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:20.184228 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:20.184239 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:23:20.184266 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:20.184266 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:20.184277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:20.184339 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:20.184669 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:20.185183 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:20.185856 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:20.186047 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:20.186097 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:20.188761 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:58171: BlockingRecv error: recv got EOF from 127.5.148.1:58171 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":120,"threads_started":1}
Aug 12 14:23:20 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:20 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:20.317279 16449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:20.311838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:53517)
0812 14:23:20.311975 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:20.311986 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:20.312050 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:20.312217 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:20.312218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:20.312432 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:20.312482 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:20.312484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:20.313124 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:20.313125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:20.313704 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:20.313706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:20.314083 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:20.314085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:20.314151 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:20.314799 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:20.314805 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:20.317183 (+ 2378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
W20260812 14:23:20.317416 16448 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:20.311919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:53517 (local address 127.5.148.62:35095)
0812 14:23:20.312012 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:20.312014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:20.312035 (+ 21us) server_negotiation.cc:407] Connection header received
0812 14:23:20.312113 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:20.312114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:20.312147 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:20.312199 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:20.312551 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:20.313051 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:20.313789 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:20.313980 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:20.314039 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:20.317333 (+ 3294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:53517: BlockingRecv error: recv got EOF from 127.5.148.2:53517 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 12 14:23:20 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:20 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:20.478340 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:20.465828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:58335)
0812 14:23:20.465998 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:20.466021 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:20.466081 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:20.466232 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:20.466233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:20.466439 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:20.466491 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:20.466493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:20.467227 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:20.467229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:20.467808 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:20.467810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:20.468237 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:20.468239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:20.468328 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:20.476293 (+ 7965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:20.476300 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:20.478259 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
W20260812 14:23:20.478482 16448 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:20.465956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:58335 (local address 127.5.148.62:35095)
0812 14:23:20.466014 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:20.466016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:20.466078 (+ 62us) server_negotiation.cc:407] Connection header received
0812 14:23:20.466130 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:20.466130 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:20.466145 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:20.466202 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:20.466593 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:20.467122 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:20.467901 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:20.468095 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:20.468157 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:20.478390 (+ 10233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:58335: BlockingRecv error: recv got EOF from 127.5.148.3:58335 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 12 14:23:21 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:21 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:21.194387 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:21.189285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:45535)
0812 14:23:21.189481 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:21.189496 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:21.189585 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:21.189747 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:21.189749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:21.189945 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:21.190004 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:21.190006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:21.190806 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:21.190807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:21.191473 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:21.191476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:21.191864 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:21.191866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:21.191930 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:21.192497 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:21.192502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:21.194284 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
W20260812 14:23:21.194463 16453 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:21.189434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:45535 (local address 127.5.148.62:35095)
0812 14:23:21.189586 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:21.189588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:21.189598 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:23:21.189651 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:21.189652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:21.189663 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:21.189725 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:21.190086 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:21.190732 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:21.191563 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:21.191784 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:21.191860 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:21.194388 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:45535: BlockingRecv error: recv got EOF from 127.5.148.1:45535 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
Aug 12 14:23:21 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:21 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:21.322465 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:21.317786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:32939)
0812 14:23:21.317954 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:21.317965 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:21.318034 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:21.318165 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:21.318166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:21.318348 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:21.318416 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:21.318418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:21.319047 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:21.319047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:21.319582 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:21.319583 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:21.319907 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:21.319908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:21.319953 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:21.320568 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:21.320574 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:21.322369 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
W20260812 14:23:21.322562 16453 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:21.317901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:32939 (local address 127.5.148.62:35095)
0812 14:23:21.317965 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:21.317967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:21.318016 (+ 49us) server_negotiation.cc:407] Connection header received
0812 14:23:21.318093 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:21.318094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:21.318104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:21.318146 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:21.318484 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:21.318981 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:21.319652 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:21.319839 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:21.319890 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:21.322479 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:32939: BlockingRecv error: recv got EOF from 127.5.148.2:32939 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 12 14:23:21 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:21 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:21.483383 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:21.478797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:52869)
0812 14:23:21.478955 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:21.478966 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:21.479026 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:21.479185 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:21.479186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:21.479367 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:21.479419 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:21.479421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:21.480054 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:21.480055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:21.480642 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:21.480644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:21.480954 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:21.480955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:21.480996 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:21.481532 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:21.481537 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:21.483304 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
W20260812 14:23:21.483525 16453 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:21.478866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:52869 (local address 127.5.148.62:35095)
0812 14:23:21.478931 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:21.478934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:21.479019 (+ 85us) server_negotiation.cc:407] Connection header received
0812 14:23:21.479110 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:21.479111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:21.479124 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:21.479166 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:21.479483 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:21.479980 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:21.480708 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:21.480888 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:21.480944 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:21.483407 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:52869: BlockingRecv error: recv got EOF from 127.5.148.3:52869 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Aug 12 14:23:22 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:22 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:22.199661 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:22.194929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:34723)
0812 14:23:22.195078 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:22.195095 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:22.195158 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:22.195346 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:22.195347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:22.195557 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:22.195610 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:22.195612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:22.196343 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:22.196344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:22.196926 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:22.196929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:22.197271 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:22.197272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:22.197314 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:22.197786 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:22.197791 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:22.199590 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
W20260812 14:23:22.199754 16457 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:22.195043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:34723 (local address 127.5.148.62:35095)
0812 14:23:22.195191 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:22.195193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:22.195201 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:23:22.195227 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:22.195228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:22.195242 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:22.195330 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:22.195705 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:22.196270 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:22.197005 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:22.197202 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:22.197260 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:22.199684 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:34723: BlockingRecv error: recv got EOF from 127.5.148.1:34723 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
Aug 12 14:23:22 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:22 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:22.328119 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:22.323191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:42567)
0812 14:23:22.323386 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:22.323399 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:22.323473 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:22.323617 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:22.323618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:22.323788 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:22.323880 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:22.323882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:22.324625 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:22.324626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:22.325283 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:22.325286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:22.325699 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:22.325700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:22.325764 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:22.326214 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:22.326219 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:22.328011 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
W20260812 14:23:22.328204 16457 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:22.323318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:42567 (local address 127.5.148.62:35095)
0812 14:23:22.323383 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:22.323385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:22.323473 (+ 88us) server_negotiation.cc:407] Connection header received
0812 14:23:22.323527 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:22.323528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:22.323541 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:22.323594 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:22.323960 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:22.324533 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:22.325384 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:22.325597 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:22.325697 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:22.328113 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:42567: BlockingRecv error: recv got EOF from 127.5.148.2:42567 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Aug 12 14:23:22 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:22 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:22.488511 16460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:22.483890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:59333)
0812 14:23:22.484033 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:22.484049 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:22.484111 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:22.484274 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:22.484275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:22.484456 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:22.484514 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:22.484516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:22.485146 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:22.485146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:22.485661 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:22.485662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:22.485970 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:22.485971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:22.486012 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:22.486547 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:22.486557 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:22.488397 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
W20260812 14:23:22.488539 16457 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:22.483966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:59333 (local address 127.5.148.62:35095)
0812 14:23:22.484036 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:22.484043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:22.484108 (+ 65us) server_negotiation.cc:407] Connection header received
0812 14:23:22.484183 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:22.484184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:22.484196 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:22.484255 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:22.484578 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:22.485080 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:22.485729 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:22.485906 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:22.485977 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:22.488480 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:59333: BlockingRecv error: recv got EOF from 127.5.148.3:59333 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 12 14:23:23 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:23 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:23.205569 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:23.200350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:41505)
0812 14:23:23.200540 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:23.200552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:23.200618 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:23.200792 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:23.200794 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:23.201082 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:23.201148 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:23.201168 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:23.202069 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:23.202070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:23.202604 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:23.202606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:23.202983 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:23.202984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:23.203025 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:23.203576 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:23.203582 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:23.205505 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":113,"threads_started":1}
W20260812 14:23:23.205710 16462 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:23.200483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:41505 (local address 127.5.148.62:35095)
0812 14:23:23.200627 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:23.200629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:23.200639 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:23:23.200668 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:23.200669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:23.200680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:23.200743 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:23.201282 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:23.201982 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:23.202699 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:23.202909 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:23.202974 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:23.205610 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:41505: BlockingRecv error: recv got EOF from 127.5.148.1:41505 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
Aug 12 14:23:23 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:23 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:23.333945 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:23.328724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:52269)
0812 14:23:23.328888 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:23.328906 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:23.328978 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:23.329130 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:23.329131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:23.329329 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:23.329384 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:23.329386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:23.330051 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:23.330053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:23.330628 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:23.330630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:23.330998 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:23.330999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:23.331053 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:23.331559 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:23.331565 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:23.333879 (+ 2314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
W20260812 14:23:23.334049 16462 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:23.328808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:52269 (local address 127.5.148.62:35095)
0812 14:23:23.328882 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:23.328884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:23.328975 (+ 91us) server_negotiation.cc:407] Connection header received
0812 14:23:23.329036 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:23.329037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:23.329048 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:23.329106 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:23.329456 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:23.329956 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:23.330713 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:23.330894 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:23.330958 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:23.333981 (+ 3023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:52269: BlockingRecv error: recv got EOF from 127.5.148.2:52269 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 12 14:23:23 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:23 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.3@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:23.493861 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:23.489041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.3:45551)
0812 14:23:23.489248 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:23.489260 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:23.489365 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:23.489560 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:23.489562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:23.489760 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:23.489809 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:23.489810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:23.490500 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:23.490501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:23.491067 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:23.491069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:23.491444 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:23.491445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:23.491508 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:23.491999 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:23.492004 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:23.493747 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":121,"threads_started":1}
W20260812 14:23:23.493903 16462 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:23.489128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:45551 (local address 127.5.148.62:35095)
0812 14:23:23.489228 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:23.489230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:23.489343 (+ 113us) server_negotiation.cc:407] Connection header received
0812 14:23:23.489458 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:23.489458 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:23.489472 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:23.489535 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:23.489892 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:23.490404 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:23.491157 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:23.491346 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:23.491438 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:23.493836 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.3:45551: BlockingRecv error: recv got EOF from 127.5.148.3:45551 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 12 14:23:24 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:24 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.1@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:24.211024 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:24.206212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.1:37177)
0812 14:23:24.206441 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:24.206452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:24.206517 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:24.206659 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:24.206660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:24.206832 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:24.206884 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:24.206885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.207570 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.207571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:24.208082 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:24.208083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.208494 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.208495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:24.208540 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:24.209044 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:24.209049 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:24.210949 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":162,"threads_started":1}
W20260812 14:23:24.211205 16467 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:24.206316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:37177 (local address 127.5.148.62:35095)
0812 14:23:24.206440 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:24.206442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:24.206526 (+ 84us) server_negotiation.cc:407] Connection header received
0812 14:23:24.206559 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:24.206560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:24.206573 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:24.206637 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:24.206962 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.207492 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.208206 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.208421 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.208478 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:24.211106 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.1:37177: BlockingRecv error: recv got EOF from 127.5.148.1:37177 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
Aug 12 14:23:24 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:23:24 dist-test-slave-8w9v krb5kdc[15251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.148.2@KRBTEST.COM for oryx/127.5.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:23:24.339677 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:24.334585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35095 (local address 127.5.148.2:50331)
0812 14:23:24.334749 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:24.334764 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:24.334830 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:24.334989 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:24.334990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:24.335162 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:24.335215 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:24.335216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.336041 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.336043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:24.336712 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:24.336715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.337067 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.337068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:24.337112 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:24.337672 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:24.337678 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:24.339602 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.148.62:35095: Server oryx/127.5.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
W20260812 14:23:24.339751 16467 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:24.334683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:50331 (local address 127.5.148.62:35095)
0812 14:23:24.334758 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:24.334760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:24.334817 (+ 57us) server_negotiation.cc:407] Connection header received
0812 14:23:24.334898 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:24.334899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:24.334910 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:24.334972 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:24.335284 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.335937 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.336797 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.336994 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.337052 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:24.339702 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.148.2:50331: BlockingRecv error: recv got EOF from 127.5.148.2:50331 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 14:23:24.379207 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 15920
I20260812 14:23:24.388836 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 16093
I20260812 14:23:24.399122 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 16246
I20260812 14:23:24.406174 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 15840
2026-08-12T14:23:24Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17692 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T14:23:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:23:24Z Disabled control of system clock
I20260812 14:23:24.439692 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:44227
--webserver_interface=127.5.148.62
--webserver_port=0
--builtin_ntp_servers=127.5.148.20:43889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:44227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 14:23:24.524784 16472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:24.524999 16472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:24.525022 16472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:24.526146 16472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:23:24.526181 16472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:24.526194 16472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:24.526207 16472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:23:24.526219 16472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:23:24.527520 16472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:43889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:44227
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:44227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:24.527715 16472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:24.527971 16472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:24.533604 16478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:24.533726 16472 server_base.cc:1034] running on GCE node
W20260812 14:23:24.533617 16479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:24.533591 16483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:24.534154 16472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:24.534528 16472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:24.535683 16472 hybrid_clock.cc:648] HybridClock initialized: now 1786544604535663 us; error 44 us; skew 500 ppm
I20260812 14:23:24.537056 16472 webserver.cc:460] Webserver started at http://127.5.148.62:33075/ using document root <none> and password file <none>
I20260812 14:23:24.537330 16472 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:24.537379 16472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:24.537557 16472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:24.538674 16472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "5b6f495637334007bfe3666b9a9b17df"
format_stamp: "Formatted at 2026-08-12 14:23:24 on dist-test-slave-8w9v"
server_key: "1f0800b2d4f34bfa3e49e3f96853dd8a"
server_key_iv: "fdae0bcc0c642942cf7cc1254f5e6ddf"
server_key_version: "encryptionkey@0"
I20260812 14:23:24.539103 16472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "5b6f495637334007bfe3666b9a9b17df"
format_stamp: "Formatted at 2026-08-12 14:23:24 on dist-test-slave-8w9v"
server_key: "1f0800b2d4f34bfa3e49e3f96853dd8a"
server_key_iv: "fdae0bcc0c642942cf7cc1254f5e6ddf"
server_key_version: "encryptionkey@0"
I20260812 14:23:24.540580 16472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260812 14:23:24.541460 16489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:24.541651 16472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:24.541754 16472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "5b6f495637334007bfe3666b9a9b17df"
format_stamp: "Formatted at 2026-08-12 14:23:24 on dist-test-slave-8w9v"
server_key: "1f0800b2d4f34bfa3e49e3f96853dd8a"
server_key_iv: "fdae0bcc0c642942cf7cc1254f5e6ddf"
server_key_version: "encryptionkey@0"
I20260812 14:23:24.541813 16472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:24.555444 16472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:24.555738 16472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:24.555848 16472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:24.560050 16472 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:44227
I20260812 14:23:24.560180 16548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:44227 every 8 connection(s)
I20260812 14:23:24.560566 16472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:23:24.561566 16549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:24.564630 16549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df: Bootstrap starting.
I20260812 14:23:24.564953 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 16472
I20260812 14:23:24.565026 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:23:24.565212 5712 external_mini_cluster.cc:1428] Setting key 35222a98fed961d01463c9d34279f7a0
I20260812 14:23:24.565464 16549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:24.565865 16549 log.cc:824] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df: Log is configured to *not* fsync() on all Append() calls
I20260812 14:23:24.566804 16549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df: No bootstrap required, opened a new log
I20260812 14:23:24.569046 16549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6f495637334007bfe3666b9a9b17df" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 44227 } }
I20260812 14:23:24.569227 16549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:24.569289 16549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6f495637334007bfe3666b9a9b17df, State: Initialized, Role: FOLLOWER
I20260812 14:23:24.569437 16549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6f495637334007bfe3666b9a9b17df" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 44227 } }
I20260812 14:23:24.569916 16549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:23:24.569962 16549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:23:24.570016 16549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:24.570871 16549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6f495637334007bfe3666b9a9b17df" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 44227 } }
I20260812 14:23:24.571000 16549 leader_election.cc:302] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b6f495637334007bfe3666b9a9b17df; no voters:
I20260812 14:23:24.571206 16549 leader_election.cc:288] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:23:24.571264 16554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:23:24.571465 16554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [term 1 LEADER]: Becoming Leader. State: Replica: 5b6f495637334007bfe3666b9a9b17df, State: Running, Role: LEADER
I20260812 14:23:24.571583 16549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:23:24.571610 16554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6f495637334007bfe3666b9a9b17df" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 44227 } }
I20260812 14:23:24.571969 16554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b6f495637334007bfe3666b9a9b17df. Latest consensus state: current_term: 1 leader_uuid: "5b6f495637334007bfe3666b9a9b17df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6f495637334007bfe3666b9a9b17df" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 44227 } } }
I20260812 14:23:24.572154 16554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [sys.catalog]: This master's current role is: LEADER
I20260812 14:23:24.572103 16556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b6f495637334007bfe3666b9a9b17df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6f495637334007bfe3666b9a9b17df" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 44227 } } }
I20260812 14:23:24.572264 16556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df [sys.catalog]: This master's current role is: LEADER
I20260812 14:23:24.572578 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:24.565786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.5.148.62:44227)
0812 14:23:24.566033 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:24.566043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:24.566080 (+ 37us) server_negotiation.cc:407] Connection header received
0812 14:23:24.566722 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:24.566733 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:24.566830 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:24.567010 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:24.567228 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.568870 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.569804 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.570511 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.570664 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:24.571459 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:24.571464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:24.571464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:23:24.571687 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:24.571733 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:23:24.571747 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:24.572023 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:24.572058 (+ 35us) server_negotiation.cc:299] Negotiation successful
0812 14:23:24.572182 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":115,"threads_started":1}
I20260812 14:23:24.573275 16563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:23:24.573894 16563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:23:24.575496 16563 catalog_manager.cc:1347] Generated new cluster ID: ec3542013d024699b2f9f19f54dcb7e4
I20260812 14:23:24.575551 16563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:23:24.581395 16563 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:23:24.581911 16563 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:23:24.586043 16563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b6f495637334007bfe3666b9a9b17df: Generated new TSK 0
I20260812 14:23:24.586230 16563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:23:24.588423 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:44227
--builtin_ntp_servers=127.5.148.20:43889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 14:23:24.691951 16576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:24.692224 16576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:24.692252 16576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:24.693454 16576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:24.693493 16576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:24.693518 16576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:23:24.694904 16576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:43889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:44227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:24.695179 16576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:24.695520 16576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:24.700904 16581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:24.701059 16582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:24.700918 16584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:24.701380 16576 server_base.cc:1034] running on GCE node
I20260812 14:23:24.701598 16576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:24.701897 16576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:24.703088 16576 hybrid_clock.cc:648] HybridClock initialized: now 1786544604703066 us; error 33 us; skew 500 ppm
I20260812 14:23:24.704772 16576 webserver.cc:460] Webserver started at http://127.5.148.1:35291/ using document root <none> and password file <none>
I20260812 14:23:24.705086 16576 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:24.705164 16576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:24.705318 16576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:24.706833 16576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "238ae25ea7b142779ffdf7918a657c67"
format_stamp: "Formatted at 2026-08-12 14:23:24 on dist-test-slave-8w9v"
server_key: "1ae5b8c55fdf445f72f59d67800c0f64"
server_key_iv: "bc5fb55e1428f321c199dafd92066ce6"
server_key_version: "encryptionkey@0"
I20260812 14:23:24.707258 16576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "238ae25ea7b142779ffdf7918a657c67"
format_stamp: "Formatted at 2026-08-12 14:23:24 on dist-test-slave-8w9v"
server_key: "1ae5b8c55fdf445f72f59d67800c0f64"
server_key_iv: "bc5fb55e1428f321c199dafd92066ce6"
server_key_version: "encryptionkey@0"
I20260812 14:23:24.709009 16576 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260812 14:23:24.709986 16591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:24.710203 16576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:24.710312 16576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "238ae25ea7b142779ffdf7918a657c67"
format_stamp: "Formatted at 2026-08-12 14:23:24 on dist-test-slave-8w9v"
server_key: "1ae5b8c55fdf445f72f59d67800c0f64"
server_key_iv: "bc5fb55e1428f321c199dafd92066ce6"
server_key_version: "encryptionkey@0"
I20260812 14:23:24.710395 16576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:24.728543 16576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:24.728933 16576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:24.729092 16576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:24.729367 16576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:24.729811 16576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:24.729857 16576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:24.729892 16576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:24.729910 16576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:24.737711 16576 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:37057
I20260812 14:23:24.737782 16726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:37057 every 8 connection(s)
I20260812 14:23:24.738271 16576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:23:24.744246 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:24.739198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:54745 (local address 127.5.148.62:44227)
0812 14:23:24.739246 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:24.739250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:24.739548 (+ 298us) server_negotiation.cc:407] Connection header received
0812 14:23:24.740187 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:24.740188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:24.740203 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:24.740264 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:24.740996 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.741541 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.742630 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.742843 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.742924 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:24.743876 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:24.743877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:24.743877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:23:24.743883 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:24.743899 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:23:24.743905 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:24.743976 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:24.744099 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 14:23:24.744158 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:23:24.744534 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:24.739154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:44227 (local address 127.5.148.1:54745)
0812 14:23:24.739412 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:24.739449 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:24.740029 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:24.740351 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:24.740355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:24.740364 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:24.740889 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:24.740893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.741655 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.741656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:24.742525 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:24.742528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.742938 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.742939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:24.743083 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:24.743714 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:23:24.743714 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:24.743747 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:23:24.743773 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:24.743978 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:24.743986 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:24.743989 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:23:24.744101 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 14:23:24.744249 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":94,"threads_started":1}
I20260812 14:23:24.745072 16728 heartbeater.cc:347] Connected to a master server at 127.5.148.62:44227
I20260812 14:23:24.745165 16728 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:24.745389 16728 heartbeater.cc:511] Master 127.5.148.62:44227 requested a full tablet report, sending...
I20260812 14:23:24.745946 16508 ts_manager.cc:194] Registered new tserver with Master: 238ae25ea7b142779ffdf7918a657c67 (127.5.148.1:37057)
I20260812 14:23:24.746021 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 16576
I20260812 14:23:24.746117 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:23:24.746302 5712 external_mini_cluster.cc:1428] Setting key 30cf92ef75f56e7558dfb74daa26254e
I20260812 14:23:24.746868 16508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.148.1:54745
I20260812 14:23:24.747459 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:44227
--builtin_ntp_servers=127.5.148.20:43889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 14:23:24.835893 16731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:24.836238 16731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:24.836272 16731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:24.837455 16731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:24.837493 16731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:24.837517 16731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:23:24.838794 16731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:43889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:44227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:24.839004 16731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:24.839267 16731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:24.844854 16736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:24.844856 16737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:24.844856 16739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:24.845120 16731 server_base.cc:1034] running on GCE node
I20260812 14:23:24.845376 16731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:24.845647 16731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:24.846820 16731 hybrid_clock.cc:648] HybridClock initialized: now 1786544604846793 us; error 36 us; skew 500 ppm
I20260812 14:23:24.848294 16731 webserver.cc:460] Webserver started at http://127.5.148.2:34197/ using document root <none> and password file <none>
I20260812 14:23:24.848554 16731 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:24.848625 16731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:24.848779 16731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:24.849938 16731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "5c938ce133b0448193ca96e1416f59d7"
format_stamp: "Formatted at 2026-08-12 14:23:24 on dist-test-slave-8w9v"
server_key: "8caf04725d07460fbe4b5353431959d5"
server_key_iv: "d5248f49985d9d5326de1777713807a0"
server_key_version: "encryptionkey@0"
I20260812 14:23:24.850354 16731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "5c938ce133b0448193ca96e1416f59d7"
format_stamp: "Formatted at 2026-08-12 14:23:24 on dist-test-slave-8w9v"
server_key: "8caf04725d07460fbe4b5353431959d5"
server_key_iv: "d5248f49985d9d5326de1777713807a0"
server_key_version: "encryptionkey@0"
I20260812 14:23:24.851716 16731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:23:24.852617 16745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:24.852815 16731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:23:24.852926 16731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "5c938ce133b0448193ca96e1416f59d7"
format_stamp: "Formatted at 2026-08-12 14:23:24 on dist-test-slave-8w9v"
server_key: "8caf04725d07460fbe4b5353431959d5"
server_key_iv: "d5248f49985d9d5326de1777713807a0"
server_key_version: "encryptionkey@0"
I20260812 14:23:24.853000 16731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:24.866806 16731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:24.867142 16731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:24.867296 16731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:24.867542 16731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:24.867894 16731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:24.867949 16731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:24.867991 16731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:24.868024 16731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:24.874481 16731 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:35865
I20260812 14:23:24.874583 16877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:35865 every 8 connection(s)
I20260812 14:23:24.874907 16731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:23:24.880966 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:24.875729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:35873 (local address 127.5.148.62:44227)
0812 14:23:24.875790 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:24.875792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:24.876192 (+ 400us) server_negotiation.cc:407] Connection header received
0812 14:23:24.876852 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:24.876854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:24.876868 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:24.876922 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:24.877643 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.878376 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.879575 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.879770 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.879837 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:24.880709 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:24.880710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:24.880711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:23:24.880715 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:24.880729 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:23:24.880734 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:24.880781 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:24.880859 (+ 78us) server_negotiation.cc:299] Negotiation successful
0812 14:23:24.880882 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:23:24.881156 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:24.875698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:44227 (local address 127.5.148.2:35873)
0812 14:23:24.875925 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:24.875960 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:24.876730 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:24.877008 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:24.877012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:24.877020 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:24.877543 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:24.877548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.878492 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.878493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:24.879490 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:24.879492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:24.879878 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:24.879878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:24.879992 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:24.880573 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:23:24.880573 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:24.880603 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:23:24.880627 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:24.880789 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:24.880793 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:24.880796 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:23:24.880865 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 14:23:24.880946 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":103,"threads_started":1}
I20260812 14:23:24.881601 16878 heartbeater.cc:347] Connected to a master server at 127.5.148.62:44227
I20260812 14:23:24.881716 16878 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:24.881932 16878 heartbeater.cc:511] Master 127.5.148.62:44227 requested a full tablet report, sending...
I20260812 14:23:24.882298 16508 ts_manager.cc:194] Registered new tserver with Master: 5c938ce133b0448193ca96e1416f59d7 (127.5.148.2:35865)
I20260812 14:23:24.882705 16508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.148.2:35873
I20260812 14:23:24.883757 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 16731
I20260812 14:23:24.883824 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:23:24.884018 5712 external_mini_cluster.cc:1428] Setting key a6852e58772d6c2594617979693373ff
I20260812 14:23:24.884961 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:44227
--builtin_ntp_servers=127.5.148.20:43889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 14:23:24.970939 16883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:24.971227 16883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:24.971269 16883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:24.972749 16883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:24.972832 16883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:24.972894 16883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:23:24.974922 16883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:43889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:44227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:24.975234 16883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:24.975575 16883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:24.980952 16889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:24.980973 16891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:24.980973 16888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:24.981308 16883 server_base.cc:1034] running on GCE node
I20260812 14:23:24.981513 16883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:24.981725 16883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:24.982873 16883 hybrid_clock.cc:648] HybridClock initialized: now 1786544604982863 us; error 38 us; skew 500 ppm
I20260812 14:23:24.984273 16883 webserver.cc:460] Webserver started at http://127.5.148.3:38397/ using document root <none> and password file <none>
I20260812 14:23:24.984541 16883 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:24.984591 16883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:24.984747 16883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:24.985890 16883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "98e990e6028b4c00a9ec35f38edd248b"
format_stamp: "Formatted at 2026-08-12 14:23:24 on dist-test-slave-8w9v"
server_key: "5cae66935115d5b30e5f670889b2f204"
server_key_iv: "b60ff7b2f1d99b8b45f64124618408b0"
server_key_version: "encryptionkey@0"
I20260812 14:23:24.986277 16883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "98e990e6028b4c00a9ec35f38edd248b"
format_stamp: "Formatted at 2026-08-12 14:23:24 on dist-test-slave-8w9v"
server_key: "5cae66935115d5b30e5f670889b2f204"
server_key_iv: "b60ff7b2f1d99b8b45f64124618408b0"
server_key_version: "encryptionkey@0"
I20260812 14:23:24.987655 16883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:23:24.988471 16897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:24.988653 16883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:24.988765 16883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "98e990e6028b4c00a9ec35f38edd248b"
format_stamp: "Formatted at 2026-08-12 14:23:24 on dist-test-slave-8w9v"
server_key: "5cae66935115d5b30e5f670889b2f204"
server_key_iv: "b60ff7b2f1d99b8b45f64124618408b0"
server_key_version: "encryptionkey@0"
I20260812 14:23:24.988843 16883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:25.021704 16883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:25.022096 16883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:25.022266 16883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:25.022535 16883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:25.022910 16883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:25.022948 16883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:25.023001 16883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:25.023031 16883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:25.029448 16883 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:43289
I20260812 14:23:25.029543 17027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:43289 every 8 connection(s)
I20260812 14:23:25.029908 16883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:23:25.031428 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 16883
I20260812 14:23:25.031536 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:23:25.031723 5712 external_mini_cluster.cc:1428] Setting key 76844cb97b3fff9924754d22a398d82e
I20260812 14:23:25.035601 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:25.030787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:40641 (local address 127.5.148.62:44227)
0812 14:23:25.030849 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:25.030851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:25.031090 (+ 239us) server_negotiation.cc:407] Connection header received
0812 14:23:25.031719 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:25.031720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:25.031734 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:25.031794 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:25.032602 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:25.033112 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:25.034147 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:25.034357 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:25.034442 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:25.035291 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:25.035293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:25.035293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:23:25.035298 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:25.035314 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:23:25.035318 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:25.035371 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:25.035487 (+ 116us) server_negotiation.cc:299] Negotiation successful
0812 14:23:25.035517 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:23:25.035873 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:25.030755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:44227 (local address 127.5.148.3:40641)
0812 14:23:25.030955 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:25.030993 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:25.031599 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:25.031878 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:25.031882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:25.031890 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:25.032505 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:25.032511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:25.033229 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:25.033230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:25.034046 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:25.034048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:25.034456 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:25.034457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:25.034600 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:25.035141 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:23:25.035142 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:25.035173 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:23:25.035198 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:25.035377 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:25.035382 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:25.035385 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:23:25.035490 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 14:23:25.035602 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260812 14:23:25.036469 17028 heartbeater.cc:347] Connected to a master server at 127.5.148.62:44227
I20260812 14:23:25.036561 17028 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:25.036760 17028 heartbeater.cc:511] Master 127.5.148.62:44227 requested a full tablet report, sending...
I20260812 14:23:25.037192 16508 ts_manager.cc:194] Registered new tserver with Master: 98e990e6028b4c00a9ec35f38edd248b (127.5.148.3:43289)
I20260812 14:23:25.037535 16508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.148.3:40641
I20260812 14:23:25.043609 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 14:23:25.045805 16552 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:25.045201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.5.148.62:44227)
0812 14:23:25.045268 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:25.045270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:25.045416 (+ 146us) server_negotiation.cc:407] Connection header received
0812 14:23:25.045475 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:25.045476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:25.045488 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:25.045559 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:25.045723 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44450: BlockingRecv error: recv got EOF from 127.0.0.1:44450 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:23:25.045810 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 16576
I20260812 14:23:25.057134 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 16731
I20260812 14:23:25.064944 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 16883
I20260812 14:23:25.073387 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 16472
2026-08-12T14:23:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (664 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T14:23:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:23:25Z Disabled control of system clock
I20260812 14:23:25.103677 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:39135
--webserver_interface=127.5.148.62
--webserver_port=0
--builtin_ntp_servers=127.5.148.20:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:39135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 14:23:25.188947 17039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:25.189179 17039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:25.189203 17039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:25.190431 17039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:23:25.190475 17039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:25.190488 17039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:25.190501 17039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:23:25.190553 17039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:23:25.191922 17039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:39135
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:39135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:25.192210 17039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:25.192529 17039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:25.197618 17047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:25.197629 17045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:25.197670 17039 server_base.cc:1034] running on GCE node
W20260812 14:23:25.197618 17044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:25.198055 17039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:25.198371 17039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:25.199519 17039 hybrid_clock.cc:648] HybridClock initialized: now 1786544605199500 us; error 42 us; skew 500 ppm
I20260812 14:23:25.200778 17039 webserver.cc:460] Webserver started at http://127.5.148.62:44307/ using document root <none> and password file <none>
I20260812 14:23:25.201041 17039 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:25.201090 17039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:25.201259 17039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:25.202374 17039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "cd70dad9dbcd4b48ad829dba83a1958e"
format_stamp: "Formatted at 2026-08-12 14:23:25 on dist-test-slave-8w9v"
server_key: "07f9cd140b375be2ee9cc3a1ce60eab7"
server_key_iv: "15245725da4a6256bb462bcbc92746a3"
server_key_version: "encryptionkey@0"
I20260812 14:23:25.202749 17039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "cd70dad9dbcd4b48ad829dba83a1958e"
format_stamp: "Formatted at 2026-08-12 14:23:25 on dist-test-slave-8w9v"
server_key: "07f9cd140b375be2ee9cc3a1ce60eab7"
server_key_iv: "15245725da4a6256bb462bcbc92746a3"
server_key_version: "encryptionkey@0"
I20260812 14:23:25.204166 17039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:23:25.204972 17054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:25.205188 17039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:25.205302 17039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "cd70dad9dbcd4b48ad829dba83a1958e"
format_stamp: "Formatted at 2026-08-12 14:23:25 on dist-test-slave-8w9v"
server_key: "07f9cd140b375be2ee9cc3a1ce60eab7"
server_key_iv: "15245725da4a6256bb462bcbc92746a3"
server_key_version: "encryptionkey@0"
I20260812 14:23:25.205400 17039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:25.208976 17039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:25.209267 17039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:25.209412 17039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:25.213555 17039 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:39135
I20260812 14:23:25.213590 17115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:39135 every 8 connection(s)
I20260812 14:23:25.213950 17039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:23:25.214663 17116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:25.217620 17116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e: Bootstrap starting.
I20260812 14:23:25.218178 17116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:25.218433 17116 log.cc:824] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:23:25.218605 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 17039
I20260812 14:23:25.218700 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:23:25.218881 5712 external_mini_cluster.cc:1428] Setting key 2dd3e73e211d71c8c4b6e98be44ac09d
I20260812 14:23:25.219201 17116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e: No bootstrap required, opened a new log
I20260812 14:23:25.221413 17116 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd70dad9dbcd4b48ad829dba83a1958e" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39135 } }
I20260812 14:23:25.221551 17116 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:25.221598 17116 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd70dad9dbcd4b48ad829dba83a1958e, State: Initialized, Role: FOLLOWER
I20260812 14:23:25.221750 17116 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd70dad9dbcd4b48ad829dba83a1958e" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39135 } }
I20260812 14:23:25.221891 17116 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:23:25.221941 17116 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:23:25.222024 17116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:25.222080 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:25.219419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35788 (local address 127.5.148.62:39135)
0812 14:23:25.219645 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:25.219653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:25.219686 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:23:25.220512 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:25.220521 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:25.220562 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:25.220712 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:25.221467 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:25.221475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:25.221475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:23:25.221479 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:25.221517 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:23:25.221527 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:25.221609 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:25.221775 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 14:23:25.221844 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260812 14:23:25.222946 17116 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd70dad9dbcd4b48ad829dba83a1958e" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39135 } }
I20260812 14:23:25.223073 17116 leader_election.cc:302] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd70dad9dbcd4b48ad829dba83a1958e; no voters:
I20260812 14:23:25.223260 17116 leader_election.cc:288] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:23:25.223300 17121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:23:25.223498 17121 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [term 1 LEADER]: Becoming Leader. State: Replica: cd70dad9dbcd4b48ad829dba83a1958e, State: Running, Role: LEADER
I20260812 14:23:25.223544 17116 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:23:25.223626 17121 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd70dad9dbcd4b48ad829dba83a1958e" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39135 } }
I20260812 14:23:25.224078 17125 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd70dad9dbcd4b48ad829dba83a1958e. Latest consensus state: current_term: 1 leader_uuid: "cd70dad9dbcd4b48ad829dba83a1958e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd70dad9dbcd4b48ad829dba83a1958e" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39135 } } }
I20260812 14:23:25.224217 17125 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [sys.catalog]: This master's current role is: LEADER
I20260812 14:23:25.224434 17124 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd70dad9dbcd4b48ad829dba83a1958e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd70dad9dbcd4b48ad829dba83a1958e" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39135 } } }
I20260812 14:23:25.224506 17124 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e [sys.catalog]: This master's current role is: LEADER
W20260812 14:23:25.225461 17142 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:23:25.225521 17142 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:23:25.225595 17133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:23:25.225790 17133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:23:25.227226 17133 catalog_manager.cc:1347] Generated new cluster ID: 0bbbecb2eabf454fa4bc9bd9974d5134
I20260812 14:23:25.227279 17133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:23:25.256793 17133 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:23:25.257025 17133 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:23:25.263866 17133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd70dad9dbcd4b48ad829dba83a1958e: Generated new TSK 0
I20260812 14:23:25.264113 17133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:23:25.270008 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:39135
--builtin_ntp_servers=127.5.148.20:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 14:23:25.356326 17145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:25.356545 17145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:25.356568 17145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:25.357728 17145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:25.357764 17145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:25.357790 17145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:23:25.359117 17145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:39135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:25.359326 17145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:25.359599 17145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:25.364928 17155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:25.364995 17152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:25.364928 17151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:25.365393 17145 server_base.cc:1034] running on GCE node
I20260812 14:23:25.365662 17145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:25.365945 17145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:25.367141 17145 hybrid_clock.cc:648] HybridClock initialized: now 1786544605367126 us; error 46 us; skew 500 ppm
I20260812 14:23:25.368582 17145 webserver.cc:460] Webserver started at http://127.5.148.1:43053/ using document root <none> and password file <none>
I20260812 14:23:25.368861 17145 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:25.368944 17145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:25.369098 17145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:25.370253 17145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "6a8961faae454976b9b80e4e59188398"
format_stamp: "Formatted at 2026-08-12 14:23:25 on dist-test-slave-8w9v"
server_key: "1315d5f1782b6c8cbf547ff6dd53ab2c"
server_key_iv: "c32e2d7e8b021296dc082225b1aee235"
server_key_version: "encryptionkey@0"
I20260812 14:23:25.370639 17145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "6a8961faae454976b9b80e4e59188398"
format_stamp: "Formatted at 2026-08-12 14:23:25 on dist-test-slave-8w9v"
server_key: "1315d5f1782b6c8cbf547ff6dd53ab2c"
server_key_iv: "c32e2d7e8b021296dc082225b1aee235"
server_key_version: "encryptionkey@0"
I20260812 14:23:25.372092 17145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:23:25.372983 17163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:25.373232 17145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:25.373392 17145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "6a8961faae454976b9b80e4e59188398"
format_stamp: "Formatted at 2026-08-12 14:23:25 on dist-test-slave-8w9v"
server_key: "1315d5f1782b6c8cbf547ff6dd53ab2c"
server_key_iv: "c32e2d7e8b021296dc082225b1aee235"
server_key_version: "encryptionkey@0"
I20260812 14:23:25.373490 17145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:25.377343 17145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:25.377662 17145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:25.377816 17145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:25.378041 17145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:25.378371 17145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:25.378425 17145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:25.378468 17145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:25.378510 17145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:25.384713 17145 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:38233
I20260812 14:23:25.384760 17294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:38233 every 8 connection(s)
I20260812 14:23:25.385138 17145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:23:25.386088 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 17145
I20260812 14:23:25.386190 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:23:25.386370 5712 external_mini_cluster.cc:1428] Setting key 393fffdb520146a6957e55dcf7798106
I20260812 14:23:25.387511 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:39135
--builtin_ntp_servers=127.5.148.20:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 14:23:25.388947 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:25.385999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:50329 (local address 127.5.148.62:39135)
0812 14:23:25.386063 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:25.386065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:25.386412 (+ 347us) server_negotiation.cc:407] Connection header received
0812 14:23:25.387078 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:25.387079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:25.387086 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:25.387158 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:25.388702 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:25.388703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:25.388704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:23:25.388708 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:25.388724 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:23:25.388728 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:25.388776 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:25.388881 (+ 105us) server_negotiation.cc:299] Negotiation successful
0812 14:23:25.388895 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:23:25.389271 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:25.385989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:39135 (local address 127.5.148.1:50329)
0812 14:23:25.386220 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:25.386270 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:25.386936 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:25.387235 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:25.387240 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:25.387251 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:25.388527 (+ 1276us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:23:25.388528 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:25.388570 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:23:25.388607 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:25.388800 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:25.388804 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:25.388808 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:23:25.388882 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 14:23:25.388950 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
I20260812 14:23:25.389846 17295 heartbeater.cc:347] Connected to a master server at 127.5.148.62:39135
I20260812 14:23:25.389951 17295 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:25.390107 17295 heartbeater.cc:511] Master 127.5.148.62:39135 requested a full tablet report, sending...
I20260812 14:23:25.390717 17074 ts_manager.cc:194] Registered new tserver with Master: 6a8961faae454976b9b80e4e59188398 (127.5.148.1:38233)
W20260812 14:23:25.473996 17298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:25.474215 17298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:25.474238 17298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:25.475457 17298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:25.475533 17298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:25.475582 17298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:23:25.477347 17298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:39135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:25.477615 17298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:25.477964 17298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:25.483191 17305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:25.483222 17304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:25.483294 17307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:25.483551 17298 server_base.cc:1034] running on GCE node
I20260812 14:23:25.483762 17298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:25.484038 17298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:25.485214 17298 hybrid_clock.cc:648] HybridClock initialized: now 1786544605485200 us; error 29 us; skew 500 ppm
I20260812 14:23:25.495800 17298 webserver.cc:460] Webserver started at http://127.5.148.2:39477/ using document root <none> and password file <none>
I20260812 14:23:25.496112 17298 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:25.496261 17298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:25.496446 17298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:25.497603 17298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "82bcebf6e28c4927b4fd7442fb70869a"
format_stamp: "Formatted at 2026-08-12 14:23:25 on dist-test-slave-8w9v"
server_key: "44415efbb079b6ed88c34515a412c5a9"
server_key_iv: "1ecf949c2ded410c72564237f585e14a"
server_key_version: "encryptionkey@0"
I20260812 14:23:25.497997 17298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "82bcebf6e28c4927b4fd7442fb70869a"
format_stamp: "Formatted at 2026-08-12 14:23:25 on dist-test-slave-8w9v"
server_key: "44415efbb079b6ed88c34515a412c5a9"
server_key_iv: "1ecf949c2ded410c72564237f585e14a"
server_key_version: "encryptionkey@0"
I20260812 14:23:25.499454 17298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:23:25.500252 17314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:25.500461 17298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:23:25.500569 17298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "82bcebf6e28c4927b4fd7442fb70869a"
format_stamp: "Formatted at 2026-08-12 14:23:25 on dist-test-slave-8w9v"
server_key: "44415efbb079b6ed88c34515a412c5a9"
server_key_iv: "1ecf949c2ded410c72564237f585e14a"
server_key_version: "encryptionkey@0"
I20260812 14:23:25.500654 17298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:25.504128 17298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:25.504446 17298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:25.504602 17298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:25.504838 17298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:25.505180 17298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:25.505235 17298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:25.505270 17298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:25.505296 17298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:25.511456 17298 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:38441
I20260812 14:23:25.511511 17449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:38441 every 8 connection(s)
I20260812 14:23:25.511844 17298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:23:25.514518 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 17298
I20260812 14:23:25.514647 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:23:25.514847 5712 external_mini_cluster.cc:1428] Setting key 6e6b74d19a539cc7a2e96f3f8e38ef83
I20260812 14:23:25.515292 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:25.512638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:55031 (local address 127.5.148.62:39135)
0812 14:23:25.512707 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:25.512709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:25.512960 (+ 251us) server_negotiation.cc:407] Connection header received
0812 14:23:25.513490 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:25.513492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:25.513500 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:25.513552 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:25.514972 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:25.514973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:25.514974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:23:25.514979 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:25.514998 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:23:25.515003 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:25.515082 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:25.515201 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 14:23:25.515220 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 14:23:25.515503 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:25.512614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:39135 (local address 127.5.148.2:55031)
0812 14:23:25.512829 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:25.512864 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:25.513368 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:25.513628 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:25.513632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:25.513648 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:25.514770 (+ 1122us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:23:25.514771 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:25.514816 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:23:25.514855 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:25.515092 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:25.515098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:25.515101 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:23:25.515192 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 14:23:25.515262 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
I20260812 14:23:25.516000 17451 heartbeater.cc:347] Connected to a master server at 127.5.148.62:39135
I20260812 14:23:25.516049 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:39135
--builtin_ntp_servers=127.5.148.20:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 14:23:25.516119 17451 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:25.516316 17451 heartbeater.cc:511] Master 127.5.148.62:39135 requested a full tablet report, sending...
I20260812 14:23:25.516760 17074 ts_manager.cc:194] Registered new tserver with Master: 82bcebf6e28c4927b4fd7442fb70869a (127.5.148.2:38441)
W20260812 14:23:25.600446 17454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:25.600701 17454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:25.600744 17454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:25.601969 17454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:25.602037 17454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:25.602083 17454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:23:25.603605 17454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:39135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:25.603878 17454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:25.604239 17454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:25.609436 17460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:25.609565 17461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:25.609436 17463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:25.609760 17454 server_base.cc:1034] running on GCE node
I20260812 14:23:25.610013 17454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:25.610276 17454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:25.611459 17454 hybrid_clock.cc:648] HybridClock initialized: now 1786544605611442 us; error 28 us; skew 500 ppm
I20260812 14:23:25.612838 17454 webserver.cc:460] Webserver started at http://127.5.148.3:42257/ using document root <none> and password file <none>
I20260812 14:23:25.613123 17454 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:25.613198 17454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:25.613361 17454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:25.614477 17454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "2074e9c0524340768358f0dad2df0e56"
format_stamp: "Formatted at 2026-08-12 14:23:25 on dist-test-slave-8w9v"
server_key: "665816afeba4f0566fc8db348a9a675f"
server_key_iv: "04660b3e4469dc4fdc62f356c23937e8"
server_key_version: "encryptionkey@0"
I20260812 14:23:25.614879 17454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "2074e9c0524340768358f0dad2df0e56"
format_stamp: "Formatted at 2026-08-12 14:23:25 on dist-test-slave-8w9v"
server_key: "665816afeba4f0566fc8db348a9a675f"
server_key_iv: "04660b3e4469dc4fdc62f356c23937e8"
server_key_version: "encryptionkey@0"
I20260812 14:23:25.616307 17454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:23:25.617192 17469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:25.617374 17454 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:25.617488 17454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "2074e9c0524340768358f0dad2df0e56"
format_stamp: "Formatted at 2026-08-12 14:23:25 on dist-test-slave-8w9v"
server_key: "665816afeba4f0566fc8db348a9a675f"
server_key_iv: "04660b3e4469dc4fdc62f356c23937e8"
server_key_version: "encryptionkey@0"
I20260812 14:23:25.617568 17454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:25.621126 17454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:25.621410 17454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:25.621557 17454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:25.621783 17454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:25.622112 17454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:25.622166 17454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:25.622210 17454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:25.622237 17454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:25.628566 17454 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:45203
I20260812 14:23:25.628619 17606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:45203 every 8 connection(s)
I20260812 14:23:25.628969 17454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:23:25.631946 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 17454
I20260812 14:23:25.632053 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:23:25.632220 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:25.629824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:47257 (local address 127.5.148.62:39135)
0812 14:23:25.629893 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:25.629895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:25.630184 (+ 289us) server_negotiation.cc:407] Connection header received
0812 14:23:25.630673 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:25.630674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:25.630681 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:25.630735 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:25.631884 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:25.631886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:25.631886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:23:25.631892 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:25.631912 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:23:25.631917 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:25.631991 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:25.632099 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 14:23:25.632145 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:23:25.632297 5712 external_mini_cluster.cc:1428] Setting key 4c723c85c18eda7c45e2f11ea0b04d75
I20260812 14:23:25.632544 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:25.629819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:39135 (local address 127.5.148.3:47257)
0812 14:23:25.630061 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:25.630096 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:25.630581 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:25.630791 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:25.630795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:25.630803 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:25.631722 (+ 919us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:23:25.631722 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:25.631752 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:23:25.631781 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:25.631989 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:25.631991 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:25.631994 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:23:25.632096 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 14:23:25.632229 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":122,"threads_started":1}
I20260812 14:23:25.633136 17607 heartbeater.cc:347] Connected to a master server at 127.5.148.62:39135
I20260812 14:23:25.633250 17607 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:25.633415 17607 heartbeater.cc:511] Master 127.5.148.62:39135 requested a full tablet report, sending...
I20260812 14:23:25.633811 17074 ts_manager.cc:194] Registered new tserver with Master: 2074e9c0524340768358f0dad2df0e56 (127.5.148.3:45203)
I20260812 14:23:25.644280 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 14:23:25.646313 17120 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:23:25.645769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.5.148.62:39135)
0812 14:23:25.645828 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:25.645829 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:25.645971 (+ 142us) server_negotiation.cc:407] Connection header received
0812 14:23:25.646020 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:25.646020 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:25.646025 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:25.646079 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:25.646216 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35802: BlockingRecv error: recv got EOF from 127.0.0.1:35802 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:23:25.646399 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 17145
I20260812 14:23:25.658380 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 17298
I20260812 14:23:25.666405 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 17454
I20260812 14:23:25.675774 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 17039
2026-08-12T14:23:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (602 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:23:25 dist-test-slave-8w9v krb5kdc[17618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:23:25 dist-test-slave-8w9v krb5kdc[17618](info): set up 2 sockets
Aug 12 14:23:25 dist-test-slave-8w9v krb5kdc[17618](info): commencing operation
krb5kdc: starting...
W20260812 14:23:26.817101 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:23:26 dist-test-slave-8w9v krb5kdc[17618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:23:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:23:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:23:26.950886 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:46691
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:42911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:46691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:23:27.036851 17632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:27.037088 17632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:27.037128 17632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:27.038287 17632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:23:27.038353 17632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:27.038378 17632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:27.038410 17632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:23:27.038439 17632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:23:27.039942 17632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:42911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:46691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:46691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:27.040246 17632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:27.040485 17632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:27.045506 17639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:27.045523 17638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:27.045680 17632 server_base.cc:1034] running on GCE node
W20260812 14:23:27.045543 17641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:27.046057 17632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:27.046411 17632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:27.047583 17632 hybrid_clock.cc:648] HybridClock initialized: now 1786544607047569 us; error 45 us; skew 500 ppm
Aug 12 14:23:27 dist-test-slave-8w9v krb5kdc[17618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544607, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:27.050191 17632 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:23:27.050599 17632 webserver.cc:460] Webserver started at http://127.5.148.62:45793/ using document root <none> and password file <none>
I20260812 14:23:27.050801 17632 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:27.050843 17632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:27.050943 17632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:27.051981 17632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "2f079e1779af44b787c7cee84d79fc7b"
format_stamp: "Formatted at 2026-08-12 14:23:27 on dist-test-slave-8w9v"
server_key: "4fcc38eade12fb501acf68c66e0a7f3b"
server_key_iv: "1d3801c28b21fa7d00063f17fe870939"
server_key_version: "encryptionkey@0"
I20260812 14:23:27.052377 17632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "2f079e1779af44b787c7cee84d79fc7b"
format_stamp: "Formatted at 2026-08-12 14:23:27 on dist-test-slave-8w9v"
server_key: "4fcc38eade12fb501acf68c66e0a7f3b"
server_key_iv: "1d3801c28b21fa7d00063f17fe870939"
server_key_version: "encryptionkey@0"
I20260812 14:23:27.053681 17632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:23:27.054359 17653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:27.054517 17632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:23:27.054608 17632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "2f079e1779af44b787c7cee84d79fc7b"
format_stamp: "Formatted at 2026-08-12 14:23:27 on dist-test-slave-8w9v"
server_key: "4fcc38eade12fb501acf68c66e0a7f3b"
server_key_iv: "1d3801c28b21fa7d00063f17fe870939"
server_key_version: "encryptionkey@0"
I20260812 14:23:27.054664 17632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:27.078197 17632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:27.080899 17632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:27.081022 17632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:27.085158 17632 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:46691
I20260812 14:23:27.085217 17719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:46691 every 8 connection(s)
I20260812 14:23:27.085582 17632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:23:27.086534 17720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:27.086892 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 17632
I20260812 14:23:27.087002 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:23:27.087200 5712 external_mini_cluster.cc:1428] Setting key 65e612c0f438d17a30e542ec44205511
I20260812 14:23:27.090260 17720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b: Bootstrap starting.
I20260812 14:23:27.091006 17720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:27.091325 17720 log.cc:824] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b: Log is configured to *not* fsync() on all Append() calls
I20260812 14:23:27.092096 17720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b: No bootstrap required, opened a new log
Aug 12 14:23:27 dist-test-slave-8w9v krb5kdc[17618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:23:27.094810 17720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f079e1779af44b787c7cee84d79fc7b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46691 } }
I20260812 14:23:27.094974 17720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:27.095039 17720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f079e1779af44b787c7cee84d79fc7b, State: Initialized, Role: FOLLOWER
I20260812 14:23:27.095202 17720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f079e1779af44b787c7cee84d79fc7b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46691 } }
I20260812 14:23:27.095294 17720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:23:27.095319 17720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:23:27.095386 17720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:27.096148 17720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f079e1779af44b787c7cee84d79fc7b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46691 } }
I20260812 14:23:27.096266 17720 leader_election.cc:302] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f079e1779af44b787c7cee84d79fc7b; no voters:
I20260812 14:23:27.096475 17720 leader_election.cc:288] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:23:27.096504 17727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:23:27.096619 17727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [term 1 LEADER]: Becoming Leader. State: Replica: 2f079e1779af44b787c7cee84d79fc7b, State: Running, Role: LEADER
I20260812 14:23:27.096705 17727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f079e1779af44b787c7cee84d79fc7b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46691 } }
I20260812 14:23:27.096760 17720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:23:27.097072 17727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f079e1779af44b787c7cee84d79fc7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f079e1779af44b787c7cee84d79fc7b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46691 } } }
I20260812 14:23:27.097246 17727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [sys.catalog]: This master's current role is: LEADER
I20260812 14:23:27.097465 17727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f079e1779af44b787c7cee84d79fc7b. Latest consensus state: current_term: 1 leader_uuid: "2f079e1779af44b787c7cee84d79fc7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f079e1779af44b787c7cee84d79fc7b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 46691 } } }
I20260812 14:23:27.097545 17727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b [sys.catalog]: This master's current role is: LEADER
I20260812 14:23:27.097846 17738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:23:27.098407 17738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:23:27.099572 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.087791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.5.148.62:46691)
0812 14:23:27.087973 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:27.087981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:27.088008 (+ 27us) server_negotiation.cc:407] Connection header received
0812 14:23:27.088571 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:27.088580 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:27.088633 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:27.088752 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:27.089625 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.090586 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.091666 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.092196 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.092378 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.095127 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:27.095132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:27.095133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:27.095147 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:27.097443 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:27.097905 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:27.097909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:27.097910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:27.098022 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:27.098255 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:27.098256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:27.098256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:27.098590 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:27.099138 (+ 548us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:27.099236 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 14:23:27.099375 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260812 14:23:27.100544 17738 catalog_manager.cc:1347] Generated new cluster ID: 8abbe7a3f6444e9cb48fc4b7a4a53265
I20260812 14:23:27.100600 17738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:23:27.108474 17738 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:23:27.109229 17738 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:23:27.112602 17738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b: Generated new TSK 0
I20260812 14:23:27.112793 17738 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:23:27.148893 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:46691
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:42911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:23:27.234838 17755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:27.235086 17755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:27.235129 17755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:27.236346 17755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:27.236413 17755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:27.236456 17755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:23:27.237941 17755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:42911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:46691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:27.238219 17755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:27.238461 17755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:27.243819 17764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:27.243808 17763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:27.243894 17755 server_base.cc:1034] running on GCE node
W20260812 14:23:27.243808 17766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:27.244251 17755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:27.244534 17755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:27.245708 17755 hybrid_clock.cc:648] HybridClock initialized: now 1786544607245685 us; error 38 us; skew 500 ppm
Aug 12 14:23:27 dist-test-slave-8w9v krb5kdc[17618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544607, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:27.248389 17755 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:23:27.248853 17755 webserver.cc:460] Webserver started at http://127.5.148.1:35127/ using document root <none> and password file <none>
I20260812 14:23:27.249121 17755 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:27.249195 17755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:27.249348 17755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:27.250440 17755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "902a58a11941410eb33bb5ae65a580f6"
format_stamp: "Formatted at 2026-08-12 14:23:27 on dist-test-slave-8w9v"
server_key: "acc08f4459bb4fc096f0b3940f8a61be"
server_key_iv: "da257248cb22537a235ac854fcf1dfb5"
server_key_version: "encryptionkey@0"
I20260812 14:23:27.250850 17755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "902a58a11941410eb33bb5ae65a580f6"
format_stamp: "Formatted at 2026-08-12 14:23:27 on dist-test-slave-8w9v"
server_key: "acc08f4459bb4fc096f0b3940f8a61be"
server_key_iv: "da257248cb22537a235ac854fcf1dfb5"
server_key_version: "encryptionkey@0"
I20260812 14:23:27.252398 17755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260812 14:23:27.253172 17773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:27.253368 17755 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:23:27.253466 17755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "902a58a11941410eb33bb5ae65a580f6"
format_stamp: "Formatted at 2026-08-12 14:23:27 on dist-test-slave-8w9v"
server_key: "acc08f4459bb4fc096f0b3940f8a61be"
server_key_iv: "da257248cb22537a235ac854fcf1dfb5"
server_key_version: "encryptionkey@0"
I20260812 14:23:27.253556 17755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:27.278479 17755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:27.280834 17755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:27.280957 17755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:27.281158 17755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:27.281533 17755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:27.281572 17755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:27.281621 17755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:27.281647 17755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:27.287810 17755 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:41857
I20260812 14:23:27.287890 17910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:41857 every 8 connection(s)
I20260812 14:23:27.288264 17755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:23:27 dist-test-slave-8w9v krb5kdc[17618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544607, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:23:27.295469 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 17755
I20260812 14:23:27.295573 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:23:27.295794 5712 external_mini_cluster.cc:1428] Setting key 86eaa56e739165eabcda99be25a04b94
I20260812 14:23:27.298730 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.289032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:33633 (local address 127.5.148.62:46691)
0812 14:23:27.289080 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:27.289083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:27.289281 (+ 198us) server_negotiation.cc:407] Connection header received
0812 14:23:27.289847 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:27.289849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:27.289864 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:27.289916 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:27.291088 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.291599 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.292711 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.292927 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.293014 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.295152 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:27.295154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:27.295154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:27.295161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:27.297071 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:27.297480 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:27.297481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:27.297482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:27.297549 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:27.297905 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:27.297906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:27.297906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:27.298039 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:27.298120 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:27.298498 (+ 378us) server_negotiation.cc:299] Negotiation successful
0812 14:23:27.298605 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:23:27.299304 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.288991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:46691 (local address 127.5.148.1:33633)
0812 14:23:27.289143 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:27.289191 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:27.289724 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:27.289988 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:27.289992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:27.290693 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:27.290981 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.290987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.291706 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.291707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.292620 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.292623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.293027 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.293028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.293152 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.293669 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:27.293678 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:27.295063 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:27.297161 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:27.297169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:27.297170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:27.297399 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:27.297620 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:27.297621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:27.297621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:27.297821 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:27.298111 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:27.298119 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:27.298294 (+ 175us) client_negotiation.cc:769] Sending connection context
0812 14:23:27.298841 (+ 547us) client_negotiation.cc:240] Negotiation successful
0812 14:23:27.299037 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260812 14:23:27.299899 17912 heartbeater.cc:347] Connected to a master server at 127.5.148.62:46691
I20260812 14:23:27.300010 17912 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:27.300239 17912 heartbeater.cc:511] Master 127.5.148.62:46691 requested a full tablet report, sending...
I20260812 14:23:27.300855 17677 ts_manager.cc:194] Registered new tserver with Master: 902a58a11941410eb33bb5ae65a580f6 (127.5.148.1:41857)
I20260812 14:23:27.301859 17677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:33633
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:23:27.326514 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:46691
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:42911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:23:27.411957 17918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:27.412254 17918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:27.412287 17918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:27.413508 17918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:27.413553 17918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:27.413580 17918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:23:27.415086 17918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:42911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:46691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:27.415360 17918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:27.415606 17918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:27.420876 17927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:27.420884 17925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:27.420892 17923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:27.421160 17918 server_base.cc:1034] running on GCE node
I20260812 14:23:27.421404 17918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:27.421667 17918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:27.422829 17918 hybrid_clock.cc:648] HybridClock initialized: now 1786544607422814 us; error 48 us; skew 500 ppm
Aug 12 14:23:27 dist-test-slave-8w9v krb5kdc[17618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544607, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:27.425648 17918 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:23:27.426129 17918 webserver.cc:460] Webserver started at http://127.5.148.2:34809/ using document root <none> and password file <none>
I20260812 14:23:27.426381 17918 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:27.426451 17918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:27.426604 17918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:27.427688 17918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "ffa07717c86e48c694f47ce0b33afab2"
format_stamp: "Formatted at 2026-08-12 14:23:27 on dist-test-slave-8w9v"
server_key: "22c779335f4bb7a4be7c81aa9cc492d9"
server_key_iv: "46dc24befd1961fc1c4e48f95448041e"
server_key_version: "encryptionkey@0"
I20260812 14:23:27.428076 17918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "ffa07717c86e48c694f47ce0b33afab2"
format_stamp: "Formatted at 2026-08-12 14:23:27 on dist-test-slave-8w9v"
server_key: "22c779335f4bb7a4be7c81aa9cc492d9"
server_key_iv: "46dc24befd1961fc1c4e48f95448041e"
server_key_version: "encryptionkey@0"
I20260812 14:23:27.429497 17918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:23:27.430264 17936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:27.430438 17918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:23:27.430552 17918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "ffa07717c86e48c694f47ce0b33afab2"
format_stamp: "Formatted at 2026-08-12 14:23:27 on dist-test-slave-8w9v"
server_key: "22c779335f4bb7a4be7c81aa9cc492d9"
server_key_iv: "46dc24befd1961fc1c4e48f95448041e"
server_key_version: "encryptionkey@0"
I20260812 14:23:27.430636 17918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:27.452651 17918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:27.455127 17918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:27.455304 17918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:27.455575 17918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:27.455922 17918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:27.455998 17918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:27.456045 17918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:27.456076 17918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:27.462904 17918 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:38923
I20260812 14:23:27.462952 18082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:38923 every 8 connection(s)
I20260812 14:23:27.463308 17918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:23:27 dist-test-slave-8w9v krb5kdc[17618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544607, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:23:27.473085 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.464102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:39069 (local address 127.5.148.62:46691)
0812 14:23:27.464208 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:27.464210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:27.464399 (+ 189us) server_negotiation.cc:407] Connection header received
0812 14:23:27.464907 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:27.464908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:27.464920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:27.464997 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:27.465971 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.466527 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.467556 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.467765 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.467843 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.470273 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:27.470275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:27.470275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:27.470282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:27.471698 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:27.472102 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:27.472104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:27.472104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:27.472238 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:27.472478 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:27.472479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:27.472479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:27.472603 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:27.472663 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:27.472912 (+ 249us) server_negotiation.cc:299] Negotiation successful
0812 14:23:27.472989 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 14:23:27.473282 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 17918
I20260812 14:23:27.473413 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.464068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:46691 (local address 127.5.148.2:39069)
0812 14:23:27.464252 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:27.464296 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:27.464816 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:27.465058 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:27.465061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:27.465655 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:27.465863 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.465868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.466620 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.466621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.467443 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.467445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.467857 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.467858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.467977 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.468616 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:27.468625 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:27.470171 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:27.471781 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:27.471789 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:27.471791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:27.472031 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:27.472304 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:27.472305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:27.472305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:27.472414 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:27.472665 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:27.472669 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:27.472795 (+ 126us) client_negotiation.cc:769] Sending connection context
0812 14:23:27.472880 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 14:23:27.473021 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
I20260812 14:23:27.473394 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:23:27.473623 5712 external_mini_cluster.cc:1428] Setting key 08ed531975619d8e9456ab80b6eeb8f3
I20260812 14:23:27.474164 18083 heartbeater.cc:347] Connected to a master server at 127.5.148.62:46691
I20260812 14:23:27.474272 18083 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:27.474496 18083 heartbeater.cc:511] Master 127.5.148.62:46691 requested a full tablet report, sending...
I20260812 14:23:27.474928 17676 ts_manager.cc:194] Registered new tserver with Master: ffa07717c86e48c694f47ce0b33afab2 (127.5.148.2:38923)
I20260812 14:23:27.475309 17676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:39069
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:23:27.503497 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:46691
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:42911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:23:27.589478 18088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:23:27.589717 18088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:23:27.589745 18088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:23:27.590914 18088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:23:27.590950 18088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:23:27.590976 18088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:23:27.592443 18088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:42911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:46691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:23:27.592638 18088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:23:27.592833 18088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:23:27.598210 18098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:27.598225 18096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:23:27.598218 18100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:23:27.598434 18088 server_base.cc:1034] running on GCE node
I20260812 14:23:27.598717 18088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:23:27.599045 18088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:23:27.600193 18088 hybrid_clock.cc:648] HybridClock initialized: now 1786544607600161 us; error 46 us; skew 500 ppm
Aug 12 14:23:27 dist-test-slave-8w9v krb5kdc[17618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544607, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:23:27.602907 18088 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:23:27.603358 18088 webserver.cc:460] Webserver started at http://127.5.148.3:45395/ using document root <none> and password file <none>
I20260812 14:23:27.603590 18088 fs_manager.cc:359] Metadata directory not provided
I20260812 14:23:27.603636 18088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:23:27.603807 18088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:23:27.604975 18088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "c78dbdf7b2ba4884a88943e33fa831fe"
format_stamp: "Formatted at 2026-08-12 14:23:27 on dist-test-slave-8w9v"
server_key: "fad0f4783376a115267885a5451119e0"
server_key_iv: "db32f5219bd478bcf8b793609a4336ec"
server_key_version: "encryptionkey@0"
I20260812 14:23:27.605346 18088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "c78dbdf7b2ba4884a88943e33fa831fe"
format_stamp: "Formatted at 2026-08-12 14:23:27 on dist-test-slave-8w9v"
server_key: "fad0f4783376a115267885a5451119e0"
server_key_iv: "db32f5219bd478bcf8b793609a4336ec"
server_key_version: "encryptionkey@0"
I20260812 14:23:27.606714 18088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:23:27.607470 18109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:27.607671 18088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:23:27.607761 18088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "c78dbdf7b2ba4884a88943e33fa831fe"
format_stamp: "Formatted at 2026-08-12 14:23:27 on dist-test-slave-8w9v"
server_key: "fad0f4783376a115267885a5451119e0"
server_key_iv: "db32f5219bd478bcf8b793609a4336ec"
server_key_version: "encryptionkey@0"
I20260812 14:23:27.607818 18088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:23:27.618746 18088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:23:27.621330 18088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:23:27.621461 18088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:23:27.621667 18088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:23:27.622011 18088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:23:27.622052 18088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:27.622076 18088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:23:27.622114 18088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:23:27.628162 18088 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:39755
I20260812 14:23:27.628237 18243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:39755 every 8 connection(s)
I20260812 14:23:27.628587 18088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:23:27.629498 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 18088
I20260812 14:23:27.629616 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:23:27.629824 5712 external_mini_cluster.cc:1428] Setting key d0fade52195c8b3f0c52af8f6f3b33ca
Aug 12 14:23:27 dist-test-slave-8w9v krb5kdc[17618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544607, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:23:27.638441 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.629430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:59083 (local address 127.5.148.62:46691)
0812 14:23:27.629505 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:27.629508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:27.629740 (+ 232us) server_negotiation.cc:407] Connection header received
0812 14:23:27.630369 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:27.630371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:27.630383 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:27.630440 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:27.631461 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.631979 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.633053 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.633256 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.633353 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.635905 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:27.635906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:27.635907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:27.635914 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:27.637213 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:27.637618 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:27.637619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:27.637620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:27.637671 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:27.637889 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:27.637890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:27.637890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:27.637993 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:27.638044 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:27.638288 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 14:23:27.638375 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 14:23:27.638748 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.629388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:46691 (local address 127.5.148.3:59083)
0812 14:23:27.629603 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:27.629650 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:27.630248 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:27.630519 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:27.630523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:27.631148 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:23:27.631353 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.631358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.632093 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.632094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.632957 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.632960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.633361 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.633362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.633564 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.634110 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:23:27.634119 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:23:27.635813 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:23:27.637307 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:27.637314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:27.637316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:27.637549 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:27.637732 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:23:27.637732 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:23:27.637733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:23:27.637836 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:23:27.638048 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:23:27.638052 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:23:27.638175 (+ 123us) client_negotiation.cc:769] Sending connection context
0812 14:23:27.638259 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 14:23:27.638387 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260812 14:23:27.639357 18244 heartbeater.cc:347] Connected to a master server at 127.5.148.62:46691
I20260812 14:23:27.639464 18244 heartbeater.cc:464] Registering TS with master...
I20260812 14:23:27.639618 18244 heartbeater.cc:511] Master 127.5.148.62:46691 requested a full tablet report, sending...
I20260812 14:23:27.640023 17676 ts_manager.cc:194] Registered new tserver with Master: c78dbdf7b2ba4884a88943e33fa831fe (127.5.148.3:39755)
I20260812 14:23:27.640477 17676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:59083
I20260812 14:23:27.641490 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:23:27.649176 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.642753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40804 (local address 127.5.148.62:46691)
0812 14:23:27.642820 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:27.642859 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:27.642873 (+ 14us) server_negotiation.cc:407] Connection header received
0812 14:23:27.642935 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:27.642936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:27.642945 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:27.642992 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:23:27.643547 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.644325 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.645147 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.645359 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.645438 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.646667 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:23:27.646668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:23:27.646669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:23:27.646676 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:23:27.647843 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:27.648278 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:27.648280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:27.648280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:27.648343 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:23:27.648559 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:23:27.648560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:23:27.648561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:23:27.648726 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:23:27.648829 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:27.648983 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 14:23:27.649071 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 14:23:27.650717 17676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:23:27.652367 17676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:23:27.659526 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.654836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:38923 (local address 127.0.0.1:60560)
0812 14:23:27.655013 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:27.655051 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:27.655120 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:27.655677 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:27.655681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:27.655683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:27.655760 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.655762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.657166 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.657167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.658351 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.658353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.659175 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.659177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.659268 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.659284 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 14:23:27.659366 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 14:23:27.659421 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":112,"threads_started":1}
I20260812 14:23:27.659539 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.655210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.5.148.2:38923)
0812 14:23:27.655449 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:27.655453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:27.655468 (+ 15us) server_negotiation.cc:407] Connection header received
0812 14:23:27.655515 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:27.655517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:27.655535 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:27.655633 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:27.655832 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.657066 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.658471 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.659075 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.659157 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.659222 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:27.659406 (+ 184us) server_negotiation.cc:299] Negotiation successful
0812 14:23:27.659453 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":152,"threads_started":1}
I20260812 14:23:27.659822 18256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.654797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:41857 (local address 127.0.0.1:52354)
0812 14:23:27.655009 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:27.655051 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:27.655119 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:27.656227 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:27.656228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:27.656229 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:27.656282 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.656283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.657529 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.657530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.658722 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.658725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.659573 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.659575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.659652 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.659658 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 14:23:27.659698 (+ 40us) client_negotiation.cc:240] Negotiation successful
0812 14:23:27.659760 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":69,"threads_started":1}
I20260812 14:23:27.660209 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.654942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.5.148.1:41857)
0812 14:23:27.655552 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:27.655556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:27.655569 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:23:27.655609 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:27.655611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:27.655626 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:27.656251 (+ 625us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:27.656340 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.657430 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.658835 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.659468 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.659895 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.659968 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:27.660000 (+ 32us) server_negotiation.cc:299] Negotiation successful
0812 14:23:27.660053 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":108,"threads_started":1}
I20260812 14:23:27.660858 17829 tablet_service.cc:1467] Processing CreateTablet for tablet 71c8269c617141f0a34f75b67f6b086a (DEFAULT_TABLE table=test-table [id=67f0aa3f227447ffb52036a7c36ea9e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:27.660876 17993 tablet_service.cc:1467] Processing CreateTablet for tablet 71c8269c617141f0a34f75b67f6b086a (DEFAULT_TABLE table=test-table [id=67f0aa3f227447ffb52036a7c36ea9e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:27.661253 17993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c8269c617141f0a34f75b67f6b086a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:27.661260 17829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c8269c617141f0a34f75b67f6b086a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:27.661341 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.655012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:39755 (local address 127.0.0.1:53182)
0812 14:23:27.655750 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:27.655761 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:27.655833 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:27.656085 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:27.656085 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:27.656087 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:27.656168 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.656169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.658810 (+ 2641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.658812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.660172 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.660175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.661114 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.661115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.661176 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.661180 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:23:27.661230 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:23:27.661269 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":105,"threads_started":1}
I20260812 14:23:27.661569 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.655246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.5.148.3:39755)
0812 14:23:27.655432 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:27.655437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:27.655907 (+ 470us) server_negotiation.cc:407] Connection header received
0812 14:23:27.655964 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:27.655966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:27.655984 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:27.656076 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:27.657571 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.658688 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.660237 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.660945 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.661020 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.661075 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:27.661450 (+ 375us) server_negotiation.cc:299] Negotiation successful
0812 14:23:27.661494 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":107,"threads_started":1}
I20260812 14:23:27.664062 18262 tablet_bootstrap.cc:492] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2: Bootstrap starting.
I20260812 14:23:27.664336 18263 tablet_bootstrap.cc:492] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6: Bootstrap starting.
I20260812 14:23:27.664835 18262 tablet_bootstrap.cc:654] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:27.665087 18263 tablet_bootstrap.cc:654] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:27.665138 18165 tablet_service.cc:1467] Processing CreateTablet for tablet 71c8269c617141f0a34f75b67f6b086a (DEFAULT_TABLE table=test-table [id=67f0aa3f227447ffb52036a7c36ea9e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:23:27.665450 18262 log.cc:824] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2: Log is configured to *not* fsync() on all Append() calls
I20260812 14:23:27.665581 18165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c8269c617141f0a34f75b67f6b086a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:23:27.665691 18263 log.cc:824] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6: Log is configured to *not* fsync() on all Append() calls
I20260812 14:23:27.666492 18263 tablet_bootstrap.cc:492] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6: No bootstrap required, opened a new log
I20260812 14:23:27.666492 18262 tablet_bootstrap.cc:492] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2: No bootstrap required, opened a new log
I20260812 14:23:27.666570 18262 ts_tablet_manager.cc:1397] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:23:27.666570 18263 ts_tablet_manager.cc:1397] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:23:27.668666 18269 tablet_bootstrap.cc:492] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe: Bootstrap starting.
I20260812 14:23:27.668651 18263 raft_consensus.cc:348] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902a58a11941410eb33bb5ae65a580f6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41857 } } peers { permanent_uuid: "ffa07717c86e48c694f47ce0b33afab2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38923 } } peers { permanent_uuid: "c78dbdf7b2ba4884a88943e33fa831fe" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39755 } }
I20260812 14:23:27.668836 18263 raft_consensus.cc:374] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:27.668886 18263 raft_consensus.cc:729] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902a58a11941410eb33bb5ae65a580f6, State: Initialized, Role: FOLLOWER
I20260812 14:23:27.668776 18262 raft_consensus.cc:348] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902a58a11941410eb33bb5ae65a580f6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41857 } } peers { permanent_uuid: "ffa07717c86e48c694f47ce0b33afab2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38923 } } peers { permanent_uuid: "c78dbdf7b2ba4884a88943e33fa831fe" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39755 } }
I20260812 14:23:27.668951 18262 raft_consensus.cc:374] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:27.669041 18262 raft_consensus.cc:729] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa07717c86e48c694f47ce0b33afab2, State: Initialized, Role: FOLLOWER
I20260812 14:23:27.669029 18263 consensus_queue.cc:260] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902a58a11941410eb33bb5ae65a580f6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41857 } } peers { permanent_uuid: "ffa07717c86e48c694f47ce0b33afab2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38923 } } peers { permanent_uuid: "c78dbdf7b2ba4884a88943e33fa831fe" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39755 } }
I20260812 14:23:27.669185 18262 consensus_queue.cc:260] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902a58a11941410eb33bb5ae65a580f6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41857 } } peers { permanent_uuid: "ffa07717c86e48c694f47ce0b33afab2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38923 } } peers { permanent_uuid: "c78dbdf7b2ba4884a88943e33fa831fe" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39755 } }
I20260812 14:23:27.669315 18263 ts_tablet_manager.cc:1428] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:23:27.669478 18262 ts_tablet_manager.cc:1428] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:23:27.669473 17912 heartbeater.cc:503] Master 127.5.148.62:46691 was elected leader, sending a full tablet report...
I20260812 14:23:27.669540 18083 heartbeater.cc:503] Master 127.5.148.62:46691 was elected leader, sending a full tablet report...
I20260812 14:23:27.670105 18269 tablet_bootstrap.cc:654] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe: Neither blocks nor log segments found. Creating new log.
I20260812 14:23:27.670889 18269 log.cc:824] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe: Log is configured to *not* fsync() on all Append() calls
I20260812 14:23:27.671667 18269 tablet_bootstrap.cc:492] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe: No bootstrap required, opened a new log
I20260812 14:23:27.671733 18269 ts_tablet_manager.cc:1397] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:23:27.673457 18269 raft_consensus.cc:348] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902a58a11941410eb33bb5ae65a580f6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41857 } } peers { permanent_uuid: "ffa07717c86e48c694f47ce0b33afab2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38923 } } peers { permanent_uuid: "c78dbdf7b2ba4884a88943e33fa831fe" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39755 } }
I20260812 14:23:27.673596 18269 raft_consensus.cc:374] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:23:27.673620 18269 raft_consensus.cc:729] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c78dbdf7b2ba4884a88943e33fa831fe, State: Initialized, Role: FOLLOWER
I20260812 14:23:27.673779 18269 consensus_queue.cc:260] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902a58a11941410eb33bb5ae65a580f6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41857 } } peers { permanent_uuid: "ffa07717c86e48c694f47ce0b33afab2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38923 } } peers { permanent_uuid: "c78dbdf7b2ba4884a88943e33fa831fe" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39755 } }
I20260812 14:23:27.674079 18269 ts_tablet_manager.cc:1428] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:23:27.674121 18244 heartbeater.cc:503] Master 127.5.148.62:46691 was elected leader, sending a full tablet report...
W20260812 14:23:27.713976 18084 tablet.cc:2369] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:23:27.719594 18275 raft_consensus.cc:482] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:23:27.719755 18275 raft_consensus.cc:504] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902a58a11941410eb33bb5ae65a580f6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41857 } } peers { permanent_uuid: "ffa07717c86e48c694f47ce0b33afab2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38923 } } peers { permanent_uuid: "c78dbdf7b2ba4884a88943e33fa831fe" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39755 } }
I20260812 14:23:27.720237 18275 leader_election.cc:288] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 902a58a11941410eb33bb5ae65a580f6 (127.5.148.1:41857), ffa07717c86e48c694f47ce0b33afab2 (127.5.148.2:38923)
I20260812 14:23:27.723130 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.720471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:41857 (local address 127.5.148.3:60589)
0812 14:23:27.720544 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:27.720564 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:27.720633 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:27.720828 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:27.720829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:27.720830 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:27.720905 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.720907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.721548 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.721549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.722284 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.722286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.722858 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.722859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.722944 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.722947 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:23:27.723014 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:23:27.723059 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 14:23:27.723130 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.720624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:60589 (local address 127.5.148.1:41857)
0812 14:23:27.720688 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:27.720690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:27.720698 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:23:27.720733 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:27.720734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:27.720744 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:27.720807 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:27.720979 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.721459 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.722380 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.722769 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.722856 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.722867 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:27.723021 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 14:23:27.723059 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 14:23:27.723277 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.720594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:53763 (local address 127.5.148.2:38923)
0812 14:23:27.720648 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:27.720650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:27.720790 (+ 140us) server_negotiation.cc:407] Connection header received
0812 14:23:27.720847 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:27.720848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:27.720868 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:27.720936 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:23:27.721096 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.721582 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.722526 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.722920 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.723007 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.723017 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:27.723179 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 14:23:27.723214 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:23:27.723299 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.720471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:38923 (local address 127.5.148.3:53763)
0812 14:23:27.720686 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:27.720696 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:23:27.720781 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:23:27.720939 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:23:27.720940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:23:27.720942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:23:27.721022 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.721024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.721676 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.721677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.722429 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:23:27.722431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.723011 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.723012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:23:27.723092 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.723096 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:23:27.723160 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 14:23:27.723200 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":19,"thread_start_us":120,"threads_started":1}
I20260812 14:23:27.723568 17855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c8269c617141f0a34f75b67f6b086a" candidate_uuid: "c78dbdf7b2ba4884a88943e33fa831fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "902a58a11941410eb33bb5ae65a580f6" is_pre_election: true
I20260812 14:23:27.723635 18017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c8269c617141f0a34f75b67f6b086a" candidate_uuid: "c78dbdf7b2ba4884a88943e33fa831fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa07717c86e48c694f47ce0b33afab2" is_pre_election: true
I20260812 14:23:27.723755 17855 raft_consensus.cc:2454] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c78dbdf7b2ba4884a88943e33fa831fe in term 0.
I20260812 14:23:27.723824 18017 raft_consensus.cc:2454] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c78dbdf7b2ba4884a88943e33fa831fe in term 0.
I20260812 14:23:27.724038 18111 leader_election.cc:302] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 902a58a11941410eb33bb5ae65a580f6, c78dbdf7b2ba4884a88943e33fa831fe; no voters:
I20260812 14:23:27.724220 18275 raft_consensus.cc:2781] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:23:27.724290 18275 raft_consensus.cc:482] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:23:27.724308 18275 raft_consensus.cc:3037] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:27.725044 18275 raft_consensus.cc:504] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902a58a11941410eb33bb5ae65a580f6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41857 } } peers { permanent_uuid: "ffa07717c86e48c694f47ce0b33afab2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38923 } } peers { permanent_uuid: "c78dbdf7b2ba4884a88943e33fa831fe" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39755 } }
I20260812 14:23:27.725196 18275 leader_election.cc:288] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [CANDIDATE]: Term 1 election: Requested vote from peers 902a58a11941410eb33bb5ae65a580f6 (127.5.148.1:41857), ffa07717c86e48c694f47ce0b33afab2 (127.5.148.2:38923)
I20260812 14:23:27.725400 18017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c8269c617141f0a34f75b67f6b086a" candidate_uuid: "c78dbdf7b2ba4884a88943e33fa831fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa07717c86e48c694f47ce0b33afab2"
I20260812 14:23:27.725419 17855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c8269c617141f0a34f75b67f6b086a" candidate_uuid: "c78dbdf7b2ba4884a88943e33fa831fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "902a58a11941410eb33bb5ae65a580f6"
I20260812 14:23:27.725497 18017 raft_consensus.cc:3037] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:27.725497 17855 raft_consensus.cc:3037] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:23:27.726482 18017 raft_consensus.cc:2454] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c78dbdf7b2ba4884a88943e33fa831fe in term 1.
I20260812 14:23:27.726482 17855 raft_consensus.cc:2454] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c78dbdf7b2ba4884a88943e33fa831fe in term 1.
I20260812 14:23:27.726668 18111 leader_election.cc:302] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 902a58a11941410eb33bb5ae65a580f6, c78dbdf7b2ba4884a88943e33fa831fe; no voters:
I20260812 14:23:27.726804 18275 raft_consensus.cc:2781] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:23:27.726976 18275 raft_consensus.cc:686] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [term 1 LEADER]: Becoming Leader. State: Replica: c78dbdf7b2ba4884a88943e33fa831fe, State: Running, Role: LEADER
I20260812 14:23:27.727083 18275 consensus_queue.cc:237] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902a58a11941410eb33bb5ae65a580f6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41857 } } peers { permanent_uuid: "ffa07717c86e48c694f47ce0b33afab2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38923 } } peers { permanent_uuid: "c78dbdf7b2ba4884a88943e33fa831fe" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39755 } }
I20260812 14:23:27.728048 17677 catalog_manager.cc:5495] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe reported cstate change: term changed from 0 to 1, leader changed from <none> to c78dbdf7b2ba4884a88943e33fa831fe (127.5.148.3). New cstate: current_term: 1 leader_uuid: "c78dbdf7b2ba4884a88943e33fa831fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902a58a11941410eb33bb5ae65a580f6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffa07717c86e48c694f47ce0b33afab2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c78dbdf7b2ba4884a88943e33fa831fe" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39755 } health_report { overall_health: HEALTHY } } }
I20260812 14:23:27.734117 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.731446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.5.148.3:39755)
0812 14:23:27.731511 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:23:27.731513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:23:27.731522 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:23:27.731596 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:23:27.731597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:23:27.731618 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:23:27.731702 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:23:27.731861 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.732430 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.733293 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:23:27.733493 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:23:27.733569 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:23:27.733742 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:23:27.733872 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:23:27.733971 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 14:23:27.733991 (+ 20us) server_negotiation.cc:299] Negotiation successful
0812 14:23:27.734012 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 14:23:27.736055 18017 raft_consensus.cc:1261] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2 [term 1 FOLLOWER]: Refusing update from remote peer c78dbdf7b2ba4884a88943e33fa831fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:27.736073 17855 raft_consensus.cc:1261] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6 [term 1 FOLLOWER]: Refusing update from remote peer c78dbdf7b2ba4884a88943e33fa831fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:23:27.736514 18277 consensus_queue.cc:1035] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "902a58a11941410eb33bb5ae65a580f6" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 41857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:27.736666 18275 consensus_queue.cc:1035] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffa07717c86e48c694f47ce0b33afab2" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 38923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:23:27.739395 18281 mvcc.cc:205] Tried to move back new op lower bound from 7317686713283600384 to 7317686713250861056. Current Snapshot: MvccSnapshot[applied={T|T < 7317686713283600384}]
I20260812 14:23:27.739784 18283 mvcc.cc:205] Tried to move back new op lower bound from 7317686713283600384 to 7317686713250861056. Current Snapshot: MvccSnapshot[applied={T|T < 7317686713283600384}]
I20260812 14:23:27.741663 17675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:23:27.741779 17675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:23:27.742681 17675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f079e1779af44b787c7cee84d79fc7b: Sending DeleteTablet for 3 replicas of tablet 71c8269c617141f0a34f75b67f6b086a
I20260812 14:23:27.743076 17829 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c8269c617141f0a34f75b67f6b086a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:27 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:52354
I20260812 14:23:27.743340 18292 tablet_replica.cc:330] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6: stopping tablet replica
I20260812 14:23:27.743363 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 17755
I20260812 14:23:27.743363 17993 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c8269c617141f0a34f75b67f6b086a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:27 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:60560
I20260812 14:23:27.743451 18292 raft_consensus.cc:2229] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:23:27.743551 18292 raft_consensus.cc:2258] T 71c8269c617141f0a34f75b67f6b086a P 902a58a11941410eb33bb5ae65a580f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:27.743588 18165 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c8269c617141f0a34f75b67f6b086a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:23:27 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:53182
I20260812 14:23:27.749228 18293 tablet_replica.cc:330] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2: stopping tablet replica
I20260812 14:23:27.749439 18294 tablet_replica.cc:330] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe: stopping tablet replica
I20260812 14:23:27.749619 18294 raft_consensus.cc:2229] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:23:27.749797 18293 raft_consensus.cc:2229] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:23:27.750012 18294 raft_consensus.cc:2258] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:27.750183 18293 raft_consensus.cc:2258] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:23:27.752242 18293 ts_tablet_manager.cc:1905] T 71c8269c617141f0a34f75b67f6b086a P ffa07717c86e48c694f47ce0b33afab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:27.752605 18294 ts_tablet_manager.cc:1905] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:23:27.755084 18294 ts_tablet_manager.cc:1918] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260812 14:23:27.755105 17655 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:41857 (error 108)
I20260812 14:23:27.755177 18294 log.cc:1196] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/71c8269c617141f0a34f75b67f6b086a
I20260812 14:23:27.755472 18294 ts_tablet_manager.cc:1939] T 71c8269c617141f0a34f75b67f6b086a P c78dbdf7b2ba4884a88943e33fa831fe: Deleting consensus metadata
I20260812 14:23:27.755777 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 17918
I20260812 14:23:27.755964 17655 catalog_manager.cc:4841] TS c78dbdf7b2ba4884a88943e33fa831fe (127.5.148.3:39755): tablet 71c8269c617141f0a34f75b67f6b086a (table test-table [id=67f0aa3f227447ffb52036a7c36ea9e0]) successfully deleted
I20260812 14:23:27.759452 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.759298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:41857 (local address 127.0.0.1:52360)
0812 14:23:27.759363 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:27.759410 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 14:23:27.759601 17655 catalog_manager.cc:4584] TS 902a58a11941410eb33bb5ae65a580f6 (127.5.148.1:41857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71c8269c617141f0a34f75b67f6b086a: Network error: Client connection negotiation failed: client connection to 127.5.148.1:41857: connect: Connection refused (error 111)
I20260812 14:23:27.771409 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:23:27.771228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:38923 (local address 127.0.0.1:60570)
0812 14:23:27.771292 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0812 14:23:27.771327 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.2:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 14:23:27.771467 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 18088
I20260812 14:23:27.780550 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 17632
2026-08-12T14:23:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2105 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:23:27 dist-test-slave-8w9v krb5kdc[18300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:23:27 dist-test-slave-8w9v krb5kdc[18300](info): set up 2 sockets
Aug 12 14:23:27 dist-test-slave-8w9v krb5kdc[18300](info): commencing operation
krb5kdc: starting...
W20260812 14:23:28.934944 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:23:29 dist-test-slave-8w9v krb5kdc[18300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:23:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:23:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/rangeradmin_127.5.148.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/rangeradmin_127.5.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/rangerlookup_127.5.148.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/rangerlookup_127.5.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/HTTP_127.5.148.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/HTTP_127.5.148.20@KRBTEST.COM.keytab.
I20260812 14:23:29.128227 5712 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAW1J0s/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/postgres -l logfile start
2026-08-12 14:23:31.930 UTC [18335] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 14:23:31.930 UTC [18335] LOG: listening on IPv4 address "127.5.148.20", port 55197
2026-08-12 14:23:31.934 UTC [18335] LOG: listening on Unix socket "/tmp/.s.PGSQL.55197"
2026-08-12 14:23:31.959 UTC [18340] LOG: database system was shut down at 2026-08-12 14:23:30 UTC
2026-08-12 14:23:31.963 UTC [18335] LOG: database system is ready to accept connections
I20260812 14:23:33.008685 5712 mini_postgres.cc:96] Postgres bound to 55197
2026-08-12 14:23:33.017 UTC [18349] FATAL: database "slave" does not exist
127.5.148.20:55197 - accepting connections
I20260812 14:23:33.017994 5712 mini_ranger.cc:161] Starting Ranger...
I20260812 14:23:33.035161 5712 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 14:23:33.578241 5712 mini_ranger.cc:87] Created ranger Postgres database
I20260812 14:23:33.578323 5712 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-admin
2026-08-12 14:23:33,910 [I] DB FLAVOR :POSTGRES
2026-08-12 14:23:33,911 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 14:23:33,911 [I] Checking connection..
2026-08-12 14:23:33,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 14:23:34,177 [I] Checking connection passed.
2026-08-12 14:23:34,177 [I] --------- Verifying version history table ---------
2026-08-12 14:23:34,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:23:34,468 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 14:23:34,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:23:34,750 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 14:23:34,750 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 14:23:34,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 14:23:35.025 UTC [18422] WARNING: there is no transaction in progress
2026-08-12 14:23:35,033 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 14:23:35,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:23:35,288 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 14:23:35,288 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 14:23:35,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 14:23:35,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 14:23:35,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8w9v.c.gcp-upstream.internal','N') ;"
2026-08-12 14:23:36,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 14:23:36,318 [I] Table x_portal_user does not exist in database ranger
2026-08-12 14:23:36,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 14:23:36,571 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 14:23:36,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 14:23:36,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 14:23:37,110 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 14:23:37,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 14:23:38.180 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:38.198 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:38.218 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:38.235 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:38.253 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:38.271 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:38.330 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:38.338 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:38.354 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:38.365 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:38.377 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:38.392 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:38.404 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:38.411 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:38.420 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:38.428 UTC [18558] WARNING: there is no transaction in progress
2026-08-12 14:23:39,118 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 14:23:39,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 14:23:39,381 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 14:23:39,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 14:23:39,642 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 14:23:39,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 14:23:39,908 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 14:23:39,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 14:23:40,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 14:23:40,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8w9v.c.gcp-upstream.internal';"
2026-08-12 14:23:40,641 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 14:23:40,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 14:23:40,889 [I] DB_PATCHES have already been applied
I20260812 14:23:40.895592 5712 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAW1J0s/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 14:23:40.895665 5712 mini_ranger.cc:193] Using host: 127.5.148.20
I20260812 14:23:40.899610 5712 mini_ranger.cc:227] Ranger admin URL: http://127.5.148.20:49443
Aug 12, 2026 2:23:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:23:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:23:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 2:23:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 2:23:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 2:23:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 2:23:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 14:23:42.137696 5712 mini_ranger.cc:160] Time spent starting Ranger: real 9.120s user 0.000s sys 0.012s
Aug 12, 2026 2:24:03 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 2:24:03 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 2:24:03 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 2:24:03 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 2:24:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 2:24:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 2:24:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 2:24:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 2:24:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 2:24:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 2:24:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 2:24:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 2:24:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 2:24:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 2:24:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 2:24:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 2:24:03 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 2:24:04 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 14:24:05.390653 5712 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/rangerkms_127.5.148.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/rangerkms_127.5.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.148.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.148.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.148.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/HTTP_127.5.148.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.148.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/HTTP_127.5.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 14:24:05 dist-test-slave-8w9v krb5kdc[18300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544645, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 14:24:05.486877 5712 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 14:24:05.498623 5712 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 14:24:06.439751 5712 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 14:24:06.439858 5712 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-kms
I20260812 14:24:06.439883 5712 mini_ranger_kms.cc:227] Using postgres at 127.5.148.20:55197
2026-08-12 14:24:07,384 [I] DB FLAVOR :POSTGRES
2026-08-12 14:24:07,384 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 14:24:07,384 [I] Checking connection
2026-08-12 14:24:07,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 14:24:07,692 [I] connection success
2026-08-12 14:24:07,692 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 14:24:07,693 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 14:24:07,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 14:24:07,986 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 14:24:07,986 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 14:24:07,986 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 14:24:07,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:55197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 14:24:08,300 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 14:24:08.849542 5712 mini_ranger_kms.cc:310] Created kms service
I20260812 14:24:09.062472 5712 mini_ranger_kms.cc:326] Created kudu user
I20260812 14:24:09.113691 5712 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 14:24:09 dist-test-slave-8w9v krb5kdc[18300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544645, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:24:09.545266 5712 mini_ranger_kms.cc:384] Added ranger policy
I20260812 14:24:09.545413 5712 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 14:24:09.545445 5712 mini_ranger_kms.cc:240] Using host: 127.5.148.20
I20260812 14:24:09.550750 5712 mini_ranger_kms.cc:276] Ranger KMS PID: 18779
I20260812 14:24:09.550913 5712 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.148.20:50195
Aug 12, 2026 2:24:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:24:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:24:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 2:24:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 2:24:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 2:24:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260812 14:24:12.067451 5712 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.581s user 0.021s sys 0.503s
I20260812 14:24:12.067571 5712 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 14:24:12.067668 5712 mini_ranger_kms.cc:401] 127.5.148.20:50195/kms/v1/keys
Aug 12 14:24:12 dist-test-slave-8w9v krb5kdc[18300](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786544652, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.148.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 14:24:12 dist-test-slave-8w9v krb5kdc[18300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544652, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.148.20@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
Aug 12 14:24:13 dist-test-slave-8w9v krb5kdc[18300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544652, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.148.20@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
Aug 12 14:24:15 dist-test-slave-8w9v krb5kdc[18300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:24:15.710114 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:43189
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:43189
--ranger_config_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.148.20:50195/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:24:15.816339 18837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:24:15.816565 18837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:24:15.816591 18837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:24:15.817771 18837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:24:15.817813 18837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:24:15.817828 18837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:24:15.817840 18837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:24:15.817852 18837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:24:15.819818 18837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.148.20:50195/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:43189
--ranger_config_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:43189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:24:15.820052 18837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:24:15.820356 18837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:24:15.826540 18845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:24:15.826502 18844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:24:15.826498 18849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:24:15.826489 18837 server_base.cc:1034] running on GCE node
I20260812 14:24:15.827011 18837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:24:15.827353 18837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:24:15.828526 18837 hybrid_clock.cc:648] HybridClock initialized: now 1786544655828513 us; error 49 us; skew 500 ppm
Aug 12 14:24:15 dist-test-slave-8w9v krb5kdc[18300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544655, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:24:15.835858 18837 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:24:15.836459 18837 webserver.cc:460] Webserver started at http://127.5.148.62:38913/ using document root <none> and password file <none>
I20260812 14:24:15.836722 18837 fs_manager.cc:359] Metadata directory not provided
I20260812 14:24:15.836787 18837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:24:15.836959 18837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:24:15 dist-test-slave-8w9v krb5kdc[18300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544655, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:24:15.897244 18837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "2e10b3e238d34896a9b9ff501af23937"
format_stamp: "Formatted at 2026-08-12 14:24:15 on dist-test-slave-8w9v"
server_key: "5092973f0af784170094bd8530093655"
server_key_iv: "040ecf3135971d921c4b125faa4172a5"
server_key_version: "kuduclusterkey@0"
I20260812 14:24:15.897722 18837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "2e10b3e238d34896a9b9ff501af23937"
format_stamp: "Formatted at 2026-08-12 14:24:15 on dist-test-slave-8w9v"
server_key: "5092973f0af784170094bd8530093655"
server_key_iv: "040ecf3135971d921c4b125faa4172a5"
server_key_version: "kuduclusterkey@0"
I20260812 14:24:15.900039 18837 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260812 14:24:15.915458 18857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:24:15.915743 18837 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:24:15.915870 18837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "2e10b3e238d34896a9b9ff501af23937"
format_stamp: "Formatted at 2026-08-12 14:24:15 on dist-test-slave-8w9v"
server_key: "5092973f0af784170094bd8530093655"
server_key_iv: "040ecf3135971d921c4b125faa4172a5"
server_key_version: "kuduclusterkey@0"
I20260812 14:24:15.915979 18837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:24:15.947947 18837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:24:15.950690 18837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:24:15.950906 18837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:24:15.958226 18837 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:43189
I20260812 14:24:15.958302 18918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:43189 every 8 connection(s)
I20260812 14:24:15.958778 18837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:24:15.959573 18919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:24:15.962334 18919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937: Bootstrap starting.
I20260812 14:24:15.963152 18919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937: Neither blocks nor log segments found. Creating new log.
I20260812 14:24:15.963747 18919 log.cc:824] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937: Log is configured to *not* fsync() on all Append() calls
I20260812 14:24:15.964665 18919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937: No bootstrap required, opened a new log
I20260812 14:24:15.967690 18919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e10b3e238d34896a9b9ff501af23937" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43189 } }
I20260812 14:24:15.967860 18919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:24:15.967885 18919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e10b3e238d34896a9b9ff501af23937, State: Initialized, Role: FOLLOWER
I20260812 14:24:15.968042 18919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e10b3e238d34896a9b9ff501af23937" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43189 } }
I20260812 14:24:15.968186 18919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:24:15.968230 18919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:24:15.968261 18919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:24:15.968860 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 18837
I20260812 14:24:15.968988 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:24:15.969151 18919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e10b3e238d34896a9b9ff501af23937" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43189 } }
I20260812 14:24:15.969206 5712 external_mini_cluster.cc:1428] Setting key 7ab8bd1520ddae3d2abe97af1a231c7f
I20260812 14:24:15.969271 18919 leader_election.cc:302] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e10b3e238d34896a9b9ff501af23937; no voters:
I20260812 14:24:15.969899 18919 leader_election.cc:288] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:24:15.969933 18925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:24:15.970151 18925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [term 1 LEADER]: Becoming Leader. State: Replica: 2e10b3e238d34896a9b9ff501af23937, State: Running, Role: LEADER
I20260812 14:24:15.970247 18925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e10b3e238d34896a9b9ff501af23937" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43189 } }
I20260812 14:24:15.970311 18919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:24:15.970623 18925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e10b3e238d34896a9b9ff501af23937. Latest consensus state: current_term: 1 leader_uuid: "2e10b3e238d34896a9b9ff501af23937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e10b3e238d34896a9b9ff501af23937" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43189 } } }
I20260812 14:24:15.970711 18925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [sys.catalog]: This master's current role is: LEADER
I20260812 14:24:15.970636 18928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e10b3e238d34896a9b9ff501af23937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e10b3e238d34896a9b9ff501af23937" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43189 } } }
I20260812 14:24:15.970788 18928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937 [sys.catalog]: This master's current role is: LEADER
I20260812 14:24:15.971028 18930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:24:15.972607 18930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:24:15.973583 18919 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Aug 12 14:24:15 dist-test-slave-8w9v krb5kdc[18300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:24:15.982165 18930 catalog_manager.cc:1347] Generated new cluster ID: b89864e861364a2d80f8c8dcb5206a75
I20260812 14:24:15.982234 18930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:24:15.987411 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:15.969747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.5.148.62:43189)
0812 14:24:15.969991 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 14:24:15.970002 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:24:15.970035 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:24:15.970195 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:24:15.970219 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:24:15.970289 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:24:15.971445 (+ 1156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:24:15.971550 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:15.973828 (+ 2278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:15.976171 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:15.978094 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:15.978358 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:15.981493 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:24:15.981498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:24:15.981498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:24:15.981512 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:24:15.984251 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:24:15.984711 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:24:15.984714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:24:15.984715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:24:15.985438 (+ 723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:24:15.985671 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:24:15.985672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:24:15.985672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:24:15.986042 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:24:15.986780 (+ 738us) server_negotiation.cc:1035] Waiting for connection context
0812 14:24:15.986872 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 14:24:15.987098 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":111,"threads_started":1}
I20260812 14:24:16.013176 18930 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:24:16.013921 18930 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:24:16.021108 18930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937: Generated new TSK 0
I20260812 14:24:16.021414 18930 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:24:16.839115 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.148.20:50195/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:43189
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:24:16.981415 18972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:24:16.981825 18972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:24:16.981927 18972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:24:16.983958 18972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:24:16.984027 18972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:24:16.984074 18972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:24:16.986701 18972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.148.20:50195/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:43189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:24:16.987001 18972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:24:16.987336 18972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:24:16.994050 18978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:24:16.994042 18980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:24:16.994283 18982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:24:16.997490 18972 server_base.cc:1034] running on GCE node
I20260812 14:24:16.997740 18972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:24:16.998106 18972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:24:16.999274 18972 hybrid_clock.cc:648] HybridClock initialized: now 1786544656999247 us; error 44 us; skew 500 ppm
Aug 12 14:24:16 dist-test-slave-8w9v krb5kdc[18300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544656, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:24:17.002918 18972 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:24:17.003578 18972 webserver.cc:460] Webserver started at http://127.5.148.1:41851/ using document root <none> and password file <none>
I20260812 14:24:17.003912 18972 fs_manager.cc:359] Metadata directory not provided
I20260812 14:24:17.004012 18972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:24:17.004252 18972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:24:17 dist-test-slave-8w9v krb5kdc[18300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544656, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:24:17.021344 18972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "5a45f6eff3d84c7c80ec72f23d56b629"
format_stamp: "Formatted at 2026-08-12 14:24:17 on dist-test-slave-8w9v"
server_key: "a33bd8121c0717fa030406a4cd2405a2"
server_key_iv: "a15b7cdb2f4e3f985e79dd8e0b21e240"
server_key_version: "kuduclusterkey@0"
I20260812 14:24:17.022015 18972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "5a45f6eff3d84c7c80ec72f23d56b629"
format_stamp: "Formatted at 2026-08-12 14:24:17 on dist-test-slave-8w9v"
server_key: "a33bd8121c0717fa030406a4cd2405a2"
server_key_iv: "a15b7cdb2f4e3f985e79dd8e0b21e240"
server_key_version: "kuduclusterkey@0"
I20260812 14:24:17.024180 18972 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260812 14:24:17.042601 18990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:24:17.042912 18972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:24:17.043113 18972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "5a45f6eff3d84c7c80ec72f23d56b629"
format_stamp: "Formatted at 2026-08-12 14:24:17 on dist-test-slave-8w9v"
server_key: "a33bd8121c0717fa030406a4cd2405a2"
server_key_iv: "a15b7cdb2f4e3f985e79dd8e0b21e240"
server_key_version: "kuduclusterkey@0"
I20260812 14:24:17.043325 18972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:24:17.088692 18972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:24:17.090852 18972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:24:17.091037 18972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:24:17.091331 18972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:24:17.091797 18972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:24:17.091871 18972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:24:17.091917 18972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:24:17.091945 18972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:24:17.100147 18972 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:32825
I20260812 14:24:17.100310 19124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:32825 every 8 connection(s)
I20260812 14:24:17.100692 18972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:24:17.104633 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 18972
I20260812 14:24:17.104759 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:24:17.104969 5712 external_mini_cluster.cc:1428] Setting key 8911f238362d3dd0292e2c8ee70e2f88
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
Aug 12 14:24:17 dist-test-slave-8w9v krb5kdc[18300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544656, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:24:17.123627 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.104899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:43189 (local address 127.5.148.1:60951)
0812 14:24:17.105207 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0812 14:24:17.105255 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 14:24:17.105953 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:24:17.108025 (+ 2072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:24:17.108031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:24:17.108373 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:24:17.108690 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.108698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.109752 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.109754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.113042 (+ 3288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.113047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.113721 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.113723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.113907 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.114617 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:24:17.114628 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:24:17.119311 (+ 4683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:24:17.121712 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:24:17.121722 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:24:17.121726 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:24:17.122085 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:24:17.122313 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:24:17.122314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:24:17.122314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:24:17.122473 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:24:17.122791 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:24:17.122798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:24:17.123000 (+ 202us) client_negotiation.cc:769] Sending connection context
0812 14:24:17.123106 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 14:24:17.123291 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":112,"threads_started":1}
I20260812 14:24:17.124464 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.104657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:60951 (local address 127.5.148.62:43189)
0812 14:24:17.107732 (+ 3075us) server_negotiation.cc:206] Beginning negotiation
0812 14:24:17.107734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:24:17.107745 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:24:17.107774 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:24:17.107775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:24:17.107790 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:24:17.107873 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:24:17.108815 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.109627 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.113279 (+ 3652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.113603 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.113696 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.119464 (+ 5768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:24:17.119466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:24:17.119466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:24:17.119476 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:24:17.121599 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:24:17.122180 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:24:17.122182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:24:17.122182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:24:17.122247 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:24:17.122546 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:24:17.122547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:24:17.122547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:24:17.122706 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:24:17.122764 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:24:17.124184 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0812 14:24:17.124293 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3045,"thread_start_us":77,"threads_started":1}
I20260812 14:24:17.125079 19125 heartbeater.cc:347] Connected to a master server at 127.5.148.62:43189
I20260812 14:24:17.125198 19125 heartbeater.cc:464] Registering TS with master...
I20260812 14:24:17.125437 19125 heartbeater.cc:511] Master 127.5.148.62:43189 requested a full tablet report, sending...
I20260812 14:24:17.126044 18877 ts_manager.cc:194] Registered new tserver with Master: 5a45f6eff3d84c7c80ec72f23d56b629 (127.5.148.1:32825)
I20260812 14:24:17.127030 18877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:60951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:24:17.137480 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.148.20:50195/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:43189
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:24:17.262919 19133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:24:17.263163 19133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:24:17.263206 19133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:24:17.264808 19133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:24:17.264881 19133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:24:17.264930 19133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:24:17.266489 19133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.148.20:50195/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:43189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:24:17.266769 19133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:24:17.267050 19133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:24:17.273337 19141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:24:17.273276 19143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:24:17.273658 19140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:24:17.273828 19133 server_base.cc:1034] running on GCE node
I20260812 14:24:17.274115 19133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:24:17.274443 19133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:24:17.275614 19133 hybrid_clock.cc:648] HybridClock initialized: now 1786544657275594 us; error 37 us; skew 500 ppm
Aug 12 14:24:17 dist-test-slave-8w9v krb5kdc[18300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:24:17.278502 19133 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:24:17.279054 19133 webserver.cc:460] Webserver started at http://127.5.148.2:34267/ using document root <none> and password file <none>
I20260812 14:24:17.279310 19133 fs_manager.cc:359] Metadata directory not provided
I20260812 14:24:17.279381 19133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:24:17.279595 19133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:24:17 dist-test-slave-8w9v krb5kdc[18300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
Aug 12 14:24:17 dist-test-slave-8w9v krb5kdc[18300](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786544657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:24:17.296610 19133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "dd6aa04126004605bba8f3d5887cc62f"
format_stamp: "Formatted at 2026-08-12 14:24:17 on dist-test-slave-8w9v"
server_key: "57ee0f4a8f944cb968bc1970bfd60f67"
server_key_iv: "4aa71b20cf9eeb2d4334a869e941a711"
server_key_version: "kuduclusterkey@0"
I20260812 14:24:17.297123 19133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "dd6aa04126004605bba8f3d5887cc62f"
format_stamp: "Formatted at 2026-08-12 14:24:17 on dist-test-slave-8w9v"
server_key: "57ee0f4a8f944cb968bc1970bfd60f67"
server_key_iv: "4aa71b20cf9eeb2d4334a869e941a711"
server_key_version: "kuduclusterkey@0"
I20260812 14:24:17.314211 19133 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.002s sys 0.001s
I20260812 14:24:17.346100 19150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:24:17.346429 19133 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260812 14:24:17.346553 19133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "dd6aa04126004605bba8f3d5887cc62f"
format_stamp: "Formatted at 2026-08-12 14:24:17 on dist-test-slave-8w9v"
server_key: "57ee0f4a8f944cb968bc1970bfd60f67"
server_key_iv: "4aa71b20cf9eeb2d4334a869e941a711"
server_key_version: "kuduclusterkey@0"
I20260812 14:24:17.346686 19133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:24:17.361045 19133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:24:17.363579 19133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:24:17.363826 19133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:24:17.364106 19133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:24:17.365293 19133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:24:17.365350 19133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:24:17.365509 19133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:24:17.365563 19133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260812 14:24:17.375731 19133 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:40477
I20260812 14:24:17.376361 19133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:24:17.376529 19284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:40477 every 8 connection(s)
I20260812 14:24:17.379675 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 19133
I20260812 14:24:17.379797 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:24:17.380000 5712 external_mini_cluster.cc:1428] Setting key 7dc42560a5be66934296335a95fc254d
Aug 12 14:24:17 dist-test-slave-8w9v krb5kdc[18300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:24:17.392923 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.380880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:57615 (local address 127.5.148.62:43189)
0812 14:24:17.380930 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:24:17.380932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:24:17.381211 (+ 279us) server_negotiation.cc:407] Connection header received
0812 14:24:17.381965 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:24:17.381966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:24:17.381979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:24:17.382011 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:24:17.383134 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.383952 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.385546 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.385916 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.386022 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.389125 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:24:17.389126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:24:17.389127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:24:17.389136 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:24:17.391101 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:24:17.391631 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:24:17.391632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:24:17.391633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:24:17.391705 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:24:17.392017 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:24:17.392018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:24:17.392018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:24:17.392221 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:24:17.392317 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 14:24:17.392679 (+ 362us) server_negotiation.cc:299] Negotiation successful
0812 14:24:17.392787 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:24:17.393029 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.380878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:43189 (local address 127.5.148.2:57615)
0812 14:24:17.381086 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 14:24:17.381133 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 14:24:17.381835 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:24:17.382209 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:24:17.382214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:24:17.382592 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:24:17.383013 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.383020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.384075 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.384076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.385445 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.385448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.386038 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.386039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.386196 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.387001 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:24:17.387012 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:24:17.389017 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:24:17.391206 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:24:17.391215 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:24:17.391218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:24:17.391543 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:24:17.391787 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:24:17.391788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:24:17.391788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:24:17.391936 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:24:17.392318 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:24:17.392326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:24:17.392519 (+ 193us) client_negotiation.cc:769] Sending connection context
0812 14:24:17.392638 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 14:24:17.392802 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":120,"threads_started":1}
I20260812 14:24:17.393961 19285 heartbeater.cc:347] Connected to a master server at 127.5.148.62:43189
I20260812 14:24:17.394068 19285 heartbeater.cc:464] Registering TS with master...
I20260812 14:24:17.394280 19285 heartbeater.cc:511] Master 127.5.148.62:43189 requested a full tablet report, sending...
I20260812 14:24:17.394678 18877 ts_manager.cc:194] Registered new tserver with Master: dd6aa04126004605bba8f3d5887cc62f (127.5.148.2:40477)
I20260812 14:24:17.395188 18877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:57615
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:24:17.419595 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.148.20:50195/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:43189
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:24:17.541862 19293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:24:17.542184 19293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:24:17.542248 19293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:24:17.544196 19293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:24:17.544263 19293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:24:17.544363 19293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:24:17.547219 19293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.148.20:50195/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:43189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:24:17.547621 19293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:24:17.547962 19293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:24:17.555796 19300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:24:17.555855 19302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:24:17.556634 19299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:24:17.557343 19293 server_base.cc:1034] running on GCE node
I20260812 14:24:17.557567 19293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:24:17.557847 19293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:24:17.559005 19293 hybrid_clock.cc:648] HybridClock initialized: now 1786544657558980 us; error 47 us; skew 500 ppm
Aug 12 14:24:17 dist-test-slave-8w9v krb5kdc[18300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:24:17.561980 19293 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:24:17.562443 19293 webserver.cc:460] Webserver started at http://127.5.148.3:45119/ using document root <none> and password file <none>
I20260812 14:24:17.562703 19293 fs_manager.cc:359] Metadata directory not provided
I20260812 14:24:17.562775 19293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:24:17.562930 19293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:24:17 dist-test-slave-8w9v krb5kdc[18300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:24:17.578051 19293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "859851a643d7459684ba080ee1c59837"
format_stamp: "Formatted at 2026-08-12 14:24:17 on dist-test-slave-8w9v"
server_key: "3851deab17fab06ceae287dc4255bb1d"
server_key_iv: "b042ad068485426820498ffe2b61f9f8"
server_key_version: "kuduclusterkey@0"
I20260812 14:24:17.578552 19293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "859851a643d7459684ba080ee1c59837"
format_stamp: "Formatted at 2026-08-12 14:24:17 on dist-test-slave-8w9v"
server_key: "3851deab17fab06ceae287dc4255bb1d"
server_key_iv: "b042ad068485426820498ffe2b61f9f8"
server_key_version: "kuduclusterkey@0"
I20260812 14:24:17.581504 19293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260812 14:24:17.601080 19311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:24:17.601369 19293 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260812 14:24:17.601459 19293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "859851a643d7459684ba080ee1c59837"
format_stamp: "Formatted at 2026-08-12 14:24:17 on dist-test-slave-8w9v"
server_key: "3851deab17fab06ceae287dc4255bb1d"
server_key_iv: "b042ad068485426820498ffe2b61f9f8"
server_key_version: "kuduclusterkey@0"
I20260812 14:24:17.601557 19293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:24:17.620477 19293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:24:17.623224 19293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:24:17.623468 19293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:24:17.623759 19293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:24:17.624346 19293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:24:17.624466 19293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:24:17.624548 19293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:24:17.624614 19293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:24:17.639312 19293 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:45373
I20260812 14:24:17.640146 19293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:24:17.647965 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 19293
I20260812 14:24:17.647922 19441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:45373 every 8 connection(s)
I20260812 14:24:17.648097 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:24:17.648427 5712 external_mini_cluster.cc:1428] Setting key 127bf4813dd09a46c0c8adf6687f9137
Aug 12 14:24:17 dist-test-slave-8w9v krb5kdc[18300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:24:17.655565 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.641537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:52557 (local address 127.5.148.62:43189)
0812 14:24:17.641631 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0812 14:24:17.641634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:24:17.642082 (+ 448us) server_negotiation.cc:407] Connection header received
0812 14:24:17.643063 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:24:17.643065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:24:17.643079 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:24:17.643176 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:24:17.644266 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.645266 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.646960 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.647294 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.647395 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.651037 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:24:17.651038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:24:17.651039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:24:17.651057 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:24:17.653453 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:24:17.654069 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:24:17.654070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:24:17.654071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:24:17.654166 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:24:17.654519 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:24:17.654520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:24:17.654520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:24:17.654729 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:24:17.654846 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 14:24:17.655280 (+ 434us) server_negotiation.cc:299] Negotiation successful
0812 14:24:17.655428 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 14:24:17.655702 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.641525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:43189 (local address 127.5.148.3:52557)
0812 14:24:17.641873 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0812 14:24:17.641926 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 14:24:17.642911 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:24:17.643278 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:24:17.643283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:24:17.643713 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:24:17.644116 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.644151 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.645397 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.645398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.646837 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.646840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.647429 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.647430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.647626 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.648464 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:24:17.648475 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:24:17.650901 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:24:17.653579 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:24:17.653590 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:24:17.653593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:24:17.653961 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:24:17.654267 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:24:17.654269 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:24:17.654269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:24:17.654432 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:24:17.654845 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:24:17.654853 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:24:17.655083 (+ 230us) client_negotiation.cc:769] Sending connection context
0812 14:24:17.655216 (+ 133us) client_negotiation.cc:240] Negotiation successful
0812 14:24:17.655400 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":204,"threads_started":1}
I20260812 14:24:17.656754 19442 heartbeater.cc:347] Connected to a master server at 127.5.148.62:43189
I20260812 14:24:17.656888 19442 heartbeater.cc:464] Registering TS with master...
I20260812 14:24:17.657111 19442 heartbeater.cc:511] Master 127.5.148.62:43189 requested a full tablet report, sending...
I20260812 14:24:17.657670 18877 ts_manager.cc:194] Registered new tserver with Master: 859851a643d7459684ba080ee1c59837 (127.5.148.3:45373)
I20260812 14:24:17.658317 18877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:52557
I20260812 14:24:17.660727 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:24:17.673840 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.663330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60470 (local address 127.5.148.62:43189)
0812 14:24:17.663395 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:24:17.663398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:24:17.663645 (+ 247us) server_negotiation.cc:407] Connection header received
0812 14:24:17.663799 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:24:17.663800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:24:17.663812 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:24:17.663853 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:24:17.664924 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.665715 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.667075 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.667531 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.667616 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.669914 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:24:17.669916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:24:17.669917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:24:17.669926 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:24:17.672200 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:24:17.672687 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:24:17.672689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:24:17.672690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:24:17.672771 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:24:17.673031 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:24:17.673032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:24:17.673032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:24:17.673204 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:24:17.673301 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 14:24:17.673593 (+ 292us) server_negotiation.cc:299] Negotiation successful
0812 14:24:17.673692 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:24:17.676647 18877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:24:17.680192 18877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:24:17.711105 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.702513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:32825 (local address 127.0.0.1:35422)
0812 14:24:17.703831 (+ 1318us) negotiation.cc:107] Waiting for socket to connect
0812 14:24:17.703840 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:24:17.703900 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:24:17.704329 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:24:17.704333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:24:17.704336 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:24:17.704427 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.704429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.706082 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.706083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.707561 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.707564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.710749 (+ 3185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.710750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.710864 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.710882 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 14:24:17.710958 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 14:24:17.711015 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1286,"thread_start_us":510,"threads_started":1}
I20260812 14:24:17.711606 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.702727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.5.148.1:32825)
0812 14:24:17.703125 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0812 14:24:17.703138 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 14:24:17.703974 (+ 836us) server_negotiation.cc:407] Connection header received
0812 14:24:17.704030 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:24:17.704042 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:24:17.704062 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:24:17.704197 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:24:17.704552 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.705966 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.707668 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.708647 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.708723 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.708799 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:24:17.711482 (+ 2683us) server_negotiation.cc:299] Negotiation successful
0812 14:24:17.711535 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":92,"threads_started":1}
I20260812 14:24:17.712683 19049 tablet_service.cc:1467] Processing CreateTablet for tablet 6401a52ec5674ca5aee8fd86feffa68d (DEFAULT_TABLE table=test-table [id=a0b6418407f74e319dcaed5b4b5af622]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:24:17.713184 19049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6401a52ec5674ca5aee8fd86feffa68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:24:17.720146 19465 tablet_bootstrap.cc:492] T 6401a52ec5674ca5aee8fd86feffa68d P 5a45f6eff3d84c7c80ec72f23d56b629: Bootstrap starting.
I20260812 14:24:17.720908 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.701462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:40477 (local address 127.0.0.1:39546)
0812 14:24:17.703228 (+ 1766us) negotiation.cc:107] Waiting for socket to connect
0812 14:24:17.703285 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 14:24:17.703478 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:24:17.709222 (+ 5744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:24:17.709223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:24:17.709225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:24:17.709273 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.709274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.718105 (+ 8831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.718105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.719263 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.719266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.720648 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.720649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.720722 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.720728 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 14:24:17.720760 (+ 32us) client_negotiation.cc:240] Negotiation successful
0812 14:24:17.720820 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1724,"thread_start_us":739,"threads_started":1}
I20260812 14:24:17.721676 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.701872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39546 (local address 127.5.148.2:40477)
0812 14:24:17.709009 (+ 7137us) server_negotiation.cc:206] Beginning negotiation
0812 14:24:17.709014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:24:17.709025 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:24:17.709084 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:24:17.709086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:24:17.709103 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:24:17.709350 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:24:17.709487 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.710654 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.719447 (+ 8793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.720537 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.721353 (+ 816us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.721512 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 14:24:17.721546 (+ 34us) server_negotiation.cc:299] Negotiation successful
0812 14:24:17.721591 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7028,"thread_start_us":161,"threads_started":1}
I20260812 14:24:17.721745 19465 tablet_bootstrap.cc:654] T 6401a52ec5674ca5aee8fd86feffa68d P 5a45f6eff3d84c7c80ec72f23d56b629: Neither blocks nor log segments found. Creating new log.
I20260812 14:24:17.722738 19465 log.cc:824] T 6401a52ec5674ca5aee8fd86feffa68d P 5a45f6eff3d84c7c80ec72f23d56b629: Log is configured to *not* fsync() on all Append() calls
I20260812 14:24:17.722723 19201 tablet_service.cc:1467] Processing CreateTablet for tablet 6401a52ec5674ca5aee8fd86feffa68d (DEFAULT_TABLE table=test-table [id=a0b6418407f74e319dcaed5b4b5af622]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:24:17.723205 19201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6401a52ec5674ca5aee8fd86feffa68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:24:17.723691 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.701482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:45373 (local address 127.0.0.1:39462)
0812 14:24:17.703648 (+ 2166us) negotiation.cc:107] Waiting for socket to connect
0812 14:24:17.703659 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:24:17.703708 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:24:17.712665 (+ 8957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:24:17.712666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:24:17.712669 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:24:17.712715 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.712717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.716207 (+ 3490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.716209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.717467 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.717472 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.723427 (+ 5955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.723428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.723500 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.723505 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:24:17.723562 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 14:24:17.723617 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2120,"thread_start_us":901,"threads_started":1}
I20260812 14:24:17.723737 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.701706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39462 (local address 127.5.148.3:45373)
0812 14:24:17.712470 (+ 10764us) server_negotiation.cc:206] Beginning negotiation
0812 14:24:17.712475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:24:17.712487 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:24:17.712528 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:24:17.712530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:24:17.712547 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:24:17.712789 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:24:17.712929 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.714263 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.718757 (+ 4494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.719757 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.723428 (+ 3671us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.723553 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 14:24:17.723590 (+ 37us) server_negotiation.cc:299] Negotiation successful
0812 14:24:17.723645 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10719,"thread_start_us":111,"threads_started":1}
I20260812 14:24:17.726305 19365 tablet_service.cc:1467] Processing CreateTablet for tablet 6401a52ec5674ca5aee8fd86feffa68d (DEFAULT_TABLE table=test-table [id=a0b6418407f74e319dcaed5b4b5af622]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:24:17.726744 19365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6401a52ec5674ca5aee8fd86feffa68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:24:17.729797 19468 tablet_bootstrap.cc:492] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f: Bootstrap starting.
I20260812 14:24:17.731438 19468 tablet_bootstrap.cc:654] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f: Neither blocks nor log segments found. Creating new log.
I20260812 14:24:17.732048 19468 log.cc:824] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f: Log is configured to *not* fsync() on all Append() calls
I20260812 14:24:17.733362 19465 tablet_bootstrap.cc:492] T 6401a52ec5674ca5aee8fd86feffa68d P 5a45f6eff3d84c7c80ec72f23d56b629: No bootstrap required, opened a new log
I20260812 14:24:17.733448 19465 ts_tablet_manager.cc:1397] T 6401a52ec5674ca5aee8fd86feffa68d P 5a45f6eff3d84c7c80ec72f23d56b629: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20260812 14:24:17.737309 19465 raft_consensus.cc:348] T 6401a52ec5674ca5aee8fd86feffa68d P 5a45f6eff3d84c7c80ec72f23d56b629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd6aa04126004605bba8f3d5887cc62f" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 40477 } } peers { permanent_uuid: "859851a643d7459684ba080ee1c59837" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 45373 } } peers { permanent_uuid: "5a45f6eff3d84c7c80ec72f23d56b629" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 32825 } }
I20260812 14:24:17.737653 19465 raft_consensus.cc:374] T 6401a52ec5674ca5aee8fd86feffa68d P 5a45f6eff3d84c7c80ec72f23d56b629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:24:17.737761 19465 raft_consensus.cc:729] T 6401a52ec5674ca5aee8fd86feffa68d P 5a45f6eff3d84c7c80ec72f23d56b629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a45f6eff3d84c7c80ec72f23d56b629, State: Initialized, Role: FOLLOWER
I20260812 14:24:17.738157 19465 consensus_queue.cc:260] T 6401a52ec5674ca5aee8fd86feffa68d P 5a45f6eff3d84c7c80ec72f23d56b629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd6aa04126004605bba8f3d5887cc62f" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 40477 } } peers { permanent_uuid: "859851a643d7459684ba080ee1c59837" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 45373 } } peers { permanent_uuid: "5a45f6eff3d84c7c80ec72f23d56b629" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 32825 } }
I20260812 14:24:17.738700 19465 ts_tablet_manager.cc:1428] T 6401a52ec5674ca5aee8fd86feffa68d P 5a45f6eff3d84c7c80ec72f23d56b629: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 14:24:17.738905 19125 heartbeater.cc:503] Master 127.5.148.62:43189 was elected leader, sending a full tablet report...
I20260812 14:24:17.740217 19469 tablet_bootstrap.cc:492] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837: Bootstrap starting.
I20260812 14:24:17.741935 19469 tablet_bootstrap.cc:654] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837: Neither blocks nor log segments found. Creating new log.
I20260812 14:24:17.742467 19468 tablet_bootstrap.cc:492] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f: No bootstrap required, opened a new log
I20260812 14:24:17.742617 19468 ts_tablet_manager.cc:1397] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20260812 14:24:17.756693 19468 raft_consensus.cc:348] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd6aa04126004605bba8f3d5887cc62f" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 40477 } } peers { permanent_uuid: "859851a643d7459684ba080ee1c59837" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 45373 } } peers { permanent_uuid: "5a45f6eff3d84c7c80ec72f23d56b629" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 32825 } }
I20260812 14:24:17.757122 19468 raft_consensus.cc:374] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:24:17.757244 19468 raft_consensus.cc:729] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6aa04126004605bba8f3d5887cc62f, State: Initialized, Role: FOLLOWER
I20260812 14:24:17.757503 19468 consensus_queue.cc:260] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd6aa04126004605bba8f3d5887cc62f" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 40477 } } peers { permanent_uuid: "859851a643d7459684ba080ee1c59837" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 45373 } } peers { permanent_uuid: "5a45f6eff3d84c7c80ec72f23d56b629" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 32825 } }
I20260812 14:24:17.758008 19468 ts_tablet_manager.cc:1428] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f: Time spent starting tablet: real 0.015s user 0.001s sys 0.004s
I20260812 14:24:17.758216 19285 heartbeater.cc:503] Master 127.5.148.62:43189 was elected leader, sending a full tablet report...
I20260812 14:24:17.760672 19469 log.cc:824] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837: Log is configured to *not* fsync() on all Append() calls
I20260812 14:24:17.763018 19469 tablet_bootstrap.cc:492] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837: No bootstrap required, opened a new log
I20260812 14:24:17.763170 19469 ts_tablet_manager.cc:1397] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20260812 14:24:17.766373 19469 raft_consensus.cc:348] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd6aa04126004605bba8f3d5887cc62f" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 40477 } } peers { permanent_uuid: "859851a643d7459684ba080ee1c59837" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 45373 } } peers { permanent_uuid: "5a45f6eff3d84c7c80ec72f23d56b629" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 32825 } }
I20260812 14:24:17.766646 19469 raft_consensus.cc:374] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:24:17.766738 19469 raft_consensus.cc:729] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859851a643d7459684ba080ee1c59837, State: Initialized, Role: FOLLOWER
I20260812 14:24:17.766953 19469 consensus_queue.cc:260] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd6aa04126004605bba8f3d5887cc62f" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 40477 } } peers { permanent_uuid: "859851a643d7459684ba080ee1c59837" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 45373 } } peers { permanent_uuid: "5a45f6eff3d84c7c80ec72f23d56b629" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 32825 } }
I20260812 14:24:17.767396 19469 ts_tablet_manager.cc:1428] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260812 14:24:17.767570 19442 heartbeater.cc:503] Master 127.5.148.62:43189 was elected leader, sending a full tablet report...
I20260812 14:24:17.824891 19472 raft_consensus.cc:482] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:24:17.825064 19472 raft_consensus.cc:504] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd6aa04126004605bba8f3d5887cc62f" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 40477 } } peers { permanent_uuid: "859851a643d7459684ba080ee1c59837" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 45373 } } peers { permanent_uuid: "5a45f6eff3d84c7c80ec72f23d56b629" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 32825 } }
I20260812 14:24:17.825578 19472 leader_election.cc:288] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 859851a643d7459684ba080ee1c59837 (127.5.148.3:45373), 5a45f6eff3d84c7c80ec72f23d56b629 (127.5.148.1:32825)
I20260812 14:24:17.830401 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.825935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:45373 (local address 127.5.148.2:40963)
0812 14:24:17.826333 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0812 14:24:17.826346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:24:17.826412 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:24:17.826756 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:24:17.826757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:24:17.826759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:24:17.826895 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.826897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.828140 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.828141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.829362 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.829366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.830110 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.830111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.830227 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.830231 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:24:17.830280 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:24:17.830322 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377}
I20260812 14:24:17.830925 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.826034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:40963 (local address 127.5.148.3:45373)
0812 14:24:17.826160 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 14:24:17.826163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:24:17.826402 (+ 239us) server_negotiation.cc:407] Connection header received
0812 14:24:17.826496 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:24:17.826497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:24:17.826522 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:24:17.826653 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:24:17.827011 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.827916 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.829477 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.830000 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.830066 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.830078 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:24:17.830818 (+ 740us) server_negotiation.cc:299] Negotiation successful
0812 14:24:17.830863 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 14:24:17.831480 19376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6401a52ec5674ca5aee8fd86feffa68d" candidate_uuid: "dd6aa04126004605bba8f3d5887cc62f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859851a643d7459684ba080ee1c59837" is_pre_election: true
I20260812 14:24:17.831651 19376 raft_consensus.cc:2454] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd6aa04126004605bba8f3d5887cc62f in term 0.
I20260812 14:24:17.832031 19155 leader_election.cc:302] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 859851a643d7459684ba080ee1c59837, dd6aa04126004605bba8f3d5887cc62f; no voters:
I20260812 14:24:17.832201 19472 raft_consensus.cc:2781] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:24:17.832247 19472 raft_consensus.cc:482] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:24:17.832269 19472 raft_consensus.cc:3037] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:24:17.833446 19472 raft_consensus.cc:504] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd6aa04126004605bba8f3d5887cc62f" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 40477 } } peers { permanent_uuid: "859851a643d7459684ba080ee1c59837" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 45373 } } peers { permanent_uuid: "5a45f6eff3d84c7c80ec72f23d56b629" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 32825 } }
I20260812 14:24:17.833595 19472 leader_election.cc:288] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [CANDIDATE]: Term 1 election: Requested vote from peers 859851a643d7459684ba080ee1c59837 (127.5.148.3:45373), 5a45f6eff3d84c7c80ec72f23d56b629 (127.5.148.1:32825)
I20260812 14:24:17.833997 19376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6401a52ec5674ca5aee8fd86feffa68d" candidate_uuid: "dd6aa04126004605bba8f3d5887cc62f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859851a643d7459684ba080ee1c59837"
I20260812 14:24:17.834086 19376 raft_consensus.cc:3037] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:24:17.835438 19376 raft_consensus.cc:2454] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd6aa04126004605bba8f3d5887cc62f in term 1.
I20260812 14:24:17.835697 19155 leader_election.cc:302] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 859851a643d7459684ba080ee1c59837, dd6aa04126004605bba8f3d5887cc62f; no voters:
I20260812 14:24:17.835835 19472 raft_consensus.cc:2781] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:24:17.836011 19472 raft_consensus.cc:686] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [term 1 LEADER]: Becoming Leader. State: Replica: dd6aa04126004605bba8f3d5887cc62f, State: Running, Role: LEADER
I20260812 14:24:17.836109 19472 consensus_queue.cc:237] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd6aa04126004605bba8f3d5887cc62f" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 40477 } } peers { permanent_uuid: "859851a643d7459684ba080ee1c59837" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 45373 } } peers { permanent_uuid: "5a45f6eff3d84c7c80ec72f23d56b629" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 32825 } }
I20260812 14:24:17.837116 18876 catalog_manager.cc:5495] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f reported cstate change: term changed from 0 to 1, leader changed from <none> to dd6aa04126004605bba8f3d5887cc62f (127.5.148.2). New cstate: current_term: 1 leader_uuid: "dd6aa04126004605bba8f3d5887cc62f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd6aa04126004605bba8f3d5887cc62f" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 40477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "859851a643d7459684ba080ee1c59837" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 45373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a45f6eff3d84c7c80ec72f23d56b629" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 32825 } health_report { overall_health: UNKNOWN } } }
I20260812 14:24:17.841109 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.826061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:32825 (local address 127.5.148.2:57153)
0812 14:24:17.830495 (+ 4434us) negotiation.cc:107] Waiting for socket to connect
0812 14:24:17.830507 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:24:17.830547 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:24:17.837886 (+ 7339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:24:17.837888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:24:17.837891 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:24:17.837939 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.837940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.838979 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.838980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.839991 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:24:17.839994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.840895 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.840896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:24:17.840953 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.840957 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:24:17.840999 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 14:24:17.841036 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4421,"thread_start_us":123,"threads_started":1}
I20260812 14:24:17.841406 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.826590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:57153 (local address 127.5.148.1:32825)
0812 14:24:17.826636 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:24:17.826638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:24:17.830673 (+ 4035us) server_negotiation.cc:407] Connection header received
0812 14:24:17.830706 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:24:17.830707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:24:17.830720 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:24:17.830763 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:24:17.838141 (+ 7378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.838826 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.840089 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.840780 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.840850 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.840862 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:24:17.841305 (+ 443us) server_negotiation.cc:299] Negotiation successful
0812 14:24:17.841348 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:24:17.841979 19070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6401a52ec5674ca5aee8fd86feffa68d" candidate_uuid: "dd6aa04126004605bba8f3d5887cc62f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a45f6eff3d84c7c80ec72f23d56b629" is_pre_election: true
I20260812 14:24:17.842135 19070 raft_consensus.cc:2454] T 6401a52ec5674ca5aee8fd86feffa68d P 5a45f6eff3d84c7c80ec72f23d56b629 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd6aa04126004605bba8f3d5887cc62f in term 0.
I20260812 14:24:17.842276 19069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6401a52ec5674ca5aee8fd86feffa68d" candidate_uuid: "dd6aa04126004605bba8f3d5887cc62f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a45f6eff3d84c7c80ec72f23d56b629"
I20260812 14:24:17.842350 19069 raft_consensus.cc:3037] T 6401a52ec5674ca5aee8fd86feffa68d P 5a45f6eff3d84c7c80ec72f23d56b629 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:24:17.843627 19069 raft_consensus.cc:2454] T 6401a52ec5674ca5aee8fd86feffa68d P 5a45f6eff3d84c7c80ec72f23d56b629 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd6aa04126004605bba8f3d5887cc62f in term 1.
W20260812 14:24:17.852226 19126 tablet.cc:2369] T 6401a52ec5674ca5aee8fd86feffa68d P 5a45f6eff3d84c7c80ec72f23d56b629: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:24:17.853490 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.848697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39560 (local address 127.5.148.2:40477)
0812 14:24:17.848738 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:24:17.848740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:24:17.848788 (+ 48us) server_negotiation.cc:407] Connection header received
0812 14:24:17.848872 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:24:17.848873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:24:17.848898 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:24:17.848983 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:24:17.849220 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.849917 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.851331 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:24:17.851802 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:24:17.851899 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:24:17.852827 (+ 928us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:24:17.852959 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:24:17.853040 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:24:17.853288 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 14:24:17.853310 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:24:17.858223 19069 raft_consensus.cc:1261] T 6401a52ec5674ca5aee8fd86feffa68d P 5a45f6eff3d84c7c80ec72f23d56b629 [term 1 FOLLOWER]: Refusing update from remote peer dd6aa04126004605bba8f3d5887cc62f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:24:17.858546 19376 raft_consensus.cc:1261] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837 [term 1 FOLLOWER]: Refusing update from remote peer dd6aa04126004605bba8f3d5887cc62f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:24:17.859057 19472 consensus_queue.cc:1035] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [LEADER]: Connected to new peer: Peer: permanent_uuid: "859851a643d7459684ba080ee1c59837" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:24:17.859202 19478 consensus_queue.cc:1035] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a45f6eff3d84c7c80ec72f23d56b629" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 32825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:24:17.869488 19482 mvcc.cc:205] Tried to move back new op lower bound from 7317686918584107008 to 7317686918497886208. Current Snapshot: MvccSnapshot[applied={T|T < 7317686918584107008}]
I20260812 14:24:17.869629 19481 mvcc.cc:205] Tried to move back new op lower bound from 7317686918584107008 to 7317686918497886208. Current Snapshot: MvccSnapshot[applied={T|T < 7317686918584107008 or (T in {7317686918584107008})}]
I20260812 14:24:17.873898 18877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:24:17.874014 18877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:24:17.876055 18877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e10b3e238d34896a9b9ff501af23937: Sending DeleteTablet for 3 replicas of tablet 6401a52ec5674ca5aee8fd86feffa68d
I20260812 14:24:17.876700 19049 tablet_service.cc:1514] Processing DeleteTablet for tablet 6401a52ec5674ca5aee8fd86feffa68d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:24:17 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:35422
I20260812 14:24:17.876803 19365 tablet_service.cc:1514] Processing DeleteTablet for tablet 6401a52ec5674ca5aee8fd86feffa68d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:24:17 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:39462
I20260812 14:24:17.877112 19201 tablet_service.cc:1514] Processing DeleteTablet for tablet 6401a52ec5674ca5aee8fd86feffa68d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:24:17 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:39546
I20260812 14:24:17.878314 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 18972
I20260812 14:24:17.891956 19493 tablet_replica.cc:330] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f: stopping tablet replica
I20260812 14:24:17.895714 19493 raft_consensus.cc:2229] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:24:17.896119 19493 raft_consensus.cc:2258] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:24:17.896862 19493 ts_tablet_manager.cc:1905] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:24:17.902369 19493 ts_tablet_manager.cc:1918] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:24:17.905102 19493 log.cc:1196] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/6401a52ec5674ca5aee8fd86feffa68d
I20260812 14:24:17.905601 19493 ts_tablet_manager.cc:1939] T 6401a52ec5674ca5aee8fd86feffa68d P dd6aa04126004605bba8f3d5887cc62f: Deleting consensus metadata
I20260812 14:24:17.906260 18858 catalog_manager.cc:4841] TS dd6aa04126004605bba8f3d5887cc62f (127.5.148.2:40477): tablet 6401a52ec5674ca5aee8fd86feffa68d (table test-table [id=a0b6418407f74e319dcaed5b4b5af622]) successfully deleted
I20260812 14:24:17.906560 19492 tablet_replica.cc:330] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837: stopping tablet replica
I20260812 14:24:17.906807 19492 raft_consensus.cc:2229] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:24:17.907043 19492 raft_consensus.cc:2258] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 12 14:24:17 dist-test-slave-8w9v krb5kdc[18300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:24:17.916333 19492 ts_tablet_manager.cc:1905] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 14:24:17.918422 18860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:32825 (error 108)
I20260812 14:24:17.919247 19492 ts_tablet_manager.cc:1918] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:24:17.919359 19492 log.cc:1196] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/6401a52ec5674ca5aee8fd86feffa68d
I20260812 14:24:17.919929 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 19133
I20260812 14:24:17.928437 19492 ts_tablet_manager.cc:1939] T 6401a52ec5674ca5aee8fd86feffa68d P 859851a643d7459684ba080ee1c59837: Deleting consensus metadata
I20260812 14:24:17.929071 18860 catalog_manager.cc:4841] TS 859851a643d7459684ba080ee1c59837 (127.5.148.3:45373): tablet 6401a52ec5674ca5aee8fd86feffa68d (table test-table [id=a0b6418407f74e319dcaed5b4b5af622]) successfully deleted
I20260812 14:24:17.929625 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:24:17.929476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:32825 (local address 127.0.0.1:35436)
0812 14:24:17.929527 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 14:24:17.929582 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260812 14:24:17.929756 18860 catalog_manager.cc:4584] TS 5a45f6eff3d84c7c80ec72f23d56b629 (127.5.148.1:32825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6401a52ec5674ca5aee8fd86feffa68d: Network error: Client connection negotiation failed: client connection to 127.5.148.1:32825: connect: Connection refused (error 111)
I20260812 14:24:17.935880 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 19293
I20260812 14:24:17.947021 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 18837
I20260812 14:24:17.959957 5712 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 14:24:18.315217 5712 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 14:24:18.315387 5712 mini_ranger.cc:66] Stopping Ranger...
I20260812 14:24:18.670902 5712 mini_ranger.cc:68] Stopped Ranger
2026-08-12 14:24:18.671 UTC [18335] LOG: received smart shutdown request
2026-08-12 14:24:18.675 UTC [18335] LOG: background worker "logical replication launcher" (PID 18346) exited with exit code 1
2026-08-12 14:24:18.676 UTC [18341] LOG: shutting down
2026-08-12 14:24:18.886 UTC [18335] LOG: database system is shut down
2026-08-12T14:24:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51228 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:24:19 dist-test-slave-8w9v krb5kdc[19504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:24:19 dist-test-slave-8w9v krb5kdc[19504](info): set up 2 sockets
Aug 12 14:24:19 dist-test-slave-8w9v krb5kdc[19504](info): commencing operation
krb5kdc: starting...
W20260812 14:24:20.149811 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:24:20 dist-test-slave-8w9v krb5kdc[19504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:24:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:24:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/rangeradmin_127.5.148.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/rangeradmin_127.5.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/rangerlookup_127.5.148.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/rangerlookup_127.5.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/HTTP_127.5.148.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/HTTP_127.5.148.20@KRBTEST.COM.keytab.
I20260812 14:24:20.332933 5712 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAW1J0s/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/postgres -l logfile start
2026-08-12 14:24:22.993 UTC [19539] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 14:24:22.993 UTC [19539] LOG: listening on IPv4 address "127.5.148.20", port 45973
2026-08-12 14:24:22.998 UTC [19539] LOG: listening on Unix socket "/tmp/.s.PGSQL.45973"
2026-08-12 14:24:23.022 UTC [19542] LOG: database system was shut down at 2026-08-12 14:24:21 UTC
2026-08-12 14:24:23.026 UTC [19539] LOG: database system is ready to accept connections
I20260812 14:24:24.093364 5712 mini_postgres.cc:96] Postgres bound to 45973
2026-08-12 14:24:24.101 UTC [19551] FATAL: database "slave" does not exist
127.5.148.20:45973 - accepting connections
I20260812 14:24:24.102043 5712 mini_ranger.cc:161] Starting Ranger...
I20260812 14:24:24.119496 5712 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 14:24:24.670110 5712 mini_ranger.cc:87] Created ranger Postgres database
I20260812 14:24:24.670221 5712 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-admin
2026-08-12 14:24:24,984 [I] DB FLAVOR :POSTGRES
2026-08-12 14:24:24,985 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 14:24:24,985 [I] Checking connection..
2026-08-12 14:24:24,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 14:24:25,270 [I] Checking connection passed.
2026-08-12 14:24:25,271 [I] --------- Verifying version history table ---------
2026-08-12 14:24:25,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:24:25,546 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 14:24:25,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:24:25,834 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 14:24:25,834 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 14:24:25,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 14:24:26.103 UTC [19625] WARNING: there is no transaction in progress
2026-08-12 14:24:26,112 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 14:24:26,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:24:26,366 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 14:24:26,366 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 14:24:26,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 14:24:26,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 14:24:26,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8w9v.c.gcp-upstream.internal','N') ;"
2026-08-12 14:24:27,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 14:24:27,443 [I] Table x_portal_user does not exist in database ranger
2026-08-12 14:24:27,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 14:24:27,737 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 14:24:27,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 14:24:28,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 14:24:28,287 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 14:24:28,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 14:24:29.318 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:29.334 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:29.352 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:29.368 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:29.386 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:29.408 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:29.470 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:29.478 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:29.493 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:29.504 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:29.515 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:29.530 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:29.541 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:29.547 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:29.555 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:29.562 UTC [19758] WARNING: there is no transaction in progress
2026-08-12 14:24:30,226 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 14:24:30,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 14:24:30,489 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 14:24:30,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 14:24:30,767 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 14:24:30,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 14:24:31,018 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 14:24:31,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 14:24:31,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 14:24:31,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8w9v.c.gcp-upstream.internal';"
2026-08-12 14:24:31,814 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 14:24:31,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 14:24:32,078 [I] DB_PATCHES have already been applied
I20260812 14:24:32.085033 5712 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAW1J0s/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 14:24:32.085108 5712 mini_ranger.cc:193] Using host: 127.5.148.20
I20260812 14:24:32.089469 5712 mini_ranger.cc:227] Ranger admin URL: http://127.5.148.20:56417
Aug 12, 2026 2:24:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:24:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:24:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 2:24:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 2:24:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 2:24:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 2:24:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 14:24:33.331400 5712 mini_ranger.cc:160] Time spent starting Ranger: real 9.229s user 0.000s sys 0.012s
Aug 12, 2026 2:24:53 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 2:24:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 2:24:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 2:24:53 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 2:24:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 2:24:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 2:24:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 2:24:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 2:24:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 2:24:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 2:24:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 2:24:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 2:24:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 2:24:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 2:24:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 2:24:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 2:24:53 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 2:24:54 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 14:24:55.552579 5712 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/rangerkms_127.5.148.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/rangerkms_127.5.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.148.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.148.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.148.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/HTTP_127.5.148.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.148.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/HTTP_127.5.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 14:24:55 dist-test-slave-8w9v krb5kdc[19504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544695, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 14:24:55.646603 5712 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 14:24:55.657640 5712 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 14:24:56.534235 5712 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 14:24:56.534336 5712 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-kms
I20260812 14:24:56.534355 5712 mini_ranger_kms.cc:227] Using postgres at 127.5.148.20:45973
2026-08-12 14:24:56,734 [I] DB FLAVOR :POSTGRES
2026-08-12 14:24:56,734 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 14:24:56,735 [I] Checking connection
2026-08-12 14:24:56,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 14:24:57,020 [I] connection success
2026-08-12 14:24:57,021 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 14:24:57,021 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 14:24:57,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 14:24:57,311 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 14:24:57,312 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 14:24:57,312 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 14:24:57,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.148.20:45973/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 14:24:57,588 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 14:24:58.103397 5712 mini_ranger_kms.cc:310] Created kms service
I20260812 14:24:58.306905 5712 mini_ranger_kms.cc:326] Created kudu user
I20260812 14:24:58.353430 5712 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 14:24:58 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544695, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:24:58.799363 5712 mini_ranger_kms.cc:384] Added ranger policy
I20260812 14:24:58.799582 5712 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAW1J0s/build/release/bin/postgresql.jar:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAW1J0s/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 14:24:58.799606 5712 mini_ranger_kms.cc:240] Using host: 127.5.148.20
I20260812 14:24:58.805148 5712 mini_ranger_kms.cc:276] Ranger KMS PID: 19976
I20260812 14:24:58.805239 5712 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.148.20:53979
Aug 12, 2026 2:24:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:24:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:24:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 2:24:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 2:25:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 2:25:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 14:25:01 dist-test-slave-8w9v krb5kdc[19504](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786544701, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.148.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:01.269201 5712 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.623s user 0.012s sys 0.172s
I20260812 14:25:01.269320 5712 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 14:25:01.269372 5712 mini_ranger_kms.cc:401] 127.5.148.20:53979/kms/v1/keys
Aug 12 14:25:01 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544701, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.148.20@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
Aug 12 14:25:01 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544701, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.148.20@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
Aug 12 14:25:04 dist-test-slave-8w9v krb5kdc[19504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:25:04.359388 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:32793
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:33153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:32793
--ranger_config_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.148.20:53979/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:04.460426 20038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:04.460670 20038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:04.460711 20038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:04.462491 20038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:25:04.462563 20038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:04.462586 20038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:04.462616 20038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:25:04.462644 20038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:25:04.464432 20038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:33153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.148.20:53979/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:32793
--ranger_config_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:32793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:04.464737 20038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:04.464993 20038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:04.470870 20046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:04.470867 20045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:04.470884 20048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:04.471218 20038 server_base.cc:1034] running on GCE node
I20260812 14:25:04.471557 20038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:04.471964 20038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:04.473157 20038 hybrid_clock.cc:648] HybridClock initialized: now 1786544704473137 us; error 36 us; skew 500 ppm
Aug 12 14:25:04 dist-test-slave-8w9v krb5kdc[19504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544704, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:04.476536 20038 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:25:04.477110 20038 webserver.cc:460] Webserver started at http://127.5.148.62:37809/ using document root <none> and password file <none>
I20260812 14:25:04.477360 20038 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:04.477401 20038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:04.477541 20038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:25:04 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544704, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:25:04.540014 20038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "eb7dacc1c2d0496b9dab9293b74734ed"
format_stamp: "Formatted at 2026-08-12 14:25:04 on dist-test-slave-8w9v"
server_key: "a808f9ba9952815cea5415d1a4d29549"
server_key_iv: "b3b85a98f65eee891fb7f268dfaad945"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:04.540525 20038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "eb7dacc1c2d0496b9dab9293b74734ed"
format_stamp: "Formatted at 2026-08-12 14:25:04 on dist-test-slave-8w9v"
server_key: "a808f9ba9952815cea5415d1a4d29549"
server_key_iv: "b3b85a98f65eee891fb7f268dfaad945"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:04.542023 20038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:25:04.554517 20059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:04.554849 20038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:25:04.554996 20038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "eb7dacc1c2d0496b9dab9293b74734ed"
format_stamp: "Formatted at 2026-08-12 14:25:04 on dist-test-slave-8w9v"
server_key: "a808f9ba9952815cea5415d1a4d29549"
server_key_iv: "b3b85a98f65eee891fb7f268dfaad945"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:04.555122 20038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:04.564675 20038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:04.566892 20038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:04.567111 20038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:04.572317 20038 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:32793
I20260812 14:25:04.572365 20122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:32793 every 8 connection(s)
I20260812 14:25:04.572803 20038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:25:04.573547 20125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:04.575951 20125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed: Bootstrap starting.
I20260812 14:25:04.576629 20125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:04.576936 20125 log.cc:824] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:04.577262 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 20038
I20260812 14:25:04.577373 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:25:04.577559 5712 external_mini_cluster.cc:1428] Setting key 8222d390b378ab76c07e3ffb8ef8bf63
I20260812 14:25:04.577692 20125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed: No bootstrap required, opened a new log
I20260812 14:25:04.580045 20125 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32793 } }
I20260812 14:25:04.580279 20125 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:04.580363 20125 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7dacc1c2d0496b9dab9293b74734ed, State: Initialized, Role: FOLLOWER
I20260812 14:25:04.580539 20125 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32793 } }
I20260812 14:25:04.580690 20125 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:25:04.580745 20125 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:25:04.580821 20125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:04.581708 20125 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32793 } }
I20260812 14:25:04.581826 20125 leader_election.cc:302] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb7dacc1c2d0496b9dab9293b74734ed; no voters:
I20260812 14:25:04.582036 20125 leader_election.cc:288] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:25:04.582105 20130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:25:04.582330 20130 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 1 LEADER]: Becoming Leader. State: Replica: eb7dacc1c2d0496b9dab9293b74734ed, State: Running, Role: LEADER
I20260812 14:25:04.582391 20125 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:25:04.582633 20130 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32793 } }
I20260812 14:25:04.583003 20132 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7dacc1c2d0496b9dab9293b74734ed. Latest consensus state: current_term: 1 leader_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32793 } } }
I20260812 14:25:04.583114 20132 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:04.583000 20131 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32793 } } }
I20260812 14:25:04.583205 20131 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:04.583396 20139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:25:04.583928 20139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:25:04.584326 20125 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Aug 12 14:25:04 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:04.590204 20139 catalog_manager.cc:1347] Generated new cluster ID: 5876acaaf5cf460197d17ddfeaf6335d
I20260812 14:25:04.590270 20139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:25:04.593634 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:04.578175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.5.148.62:32793)
0812 14:25:04.578359 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:04.578368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:04.578400 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:25:04.580065 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:04.580077 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:04.580183 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:04.580350 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:04.581607 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:04.583059 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:04.584320 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:04.584960 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:04.585070 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:04.589001 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:04.589041 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:04.589042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:04.589165 (+ 123us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:04.591548 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:04.591953 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:04.591956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:04.591957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:04.592063 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:04.592273 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:04.592273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:04.592274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:04.592592 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:04.592733 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:04.593234 (+ 501us) server_negotiation.cc:299] Negotiation successful
0812 14:25:04.593397 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260812 14:25:04.595695 20139 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:25:04.596365 20139 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:25:04.599885 20139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed: Generated new TSK 0
I20260812 14:25:04.600071 20139 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:25:05.365579 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.148.20:53979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:32793
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:33153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:05.471541 20178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:05.471750 20178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:05.471774 20178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:05.473091 20178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:05.473135 20178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:05.473162 20178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:25:05.474793 20178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:33153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.148.20:53979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:32793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:05.475034 20178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:05.475296 20178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:05.480721 20184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:05.480846 20187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:05.480721 20185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:05.481349 20178 server_base.cc:1034] running on GCE node
I20260812 14:25:05.481600 20178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:05.481860 20178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:05.483031 20178 hybrid_clock.cc:648] HybridClock initialized: now 1786544705483006 us; error 45 us; skew 500 ppm
Aug 12 14:25:05 dist-test-slave-8w9v krb5kdc[19504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544705, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:05.486166 20178 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:25:05.486639 20178 webserver.cc:460] Webserver started at http://127.5.148.1:42021/ using document root <none> and password file <none>
I20260812 14:25:05.486918 20178 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:05.486989 20178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:05.487135 20178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:25:05 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544705, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:25:05.507988 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "05fd30eadf2748c98f2692df82ec4b51"
format_stamp: "Formatted at 2026-08-12 14:25:05 on dist-test-slave-8w9v"
server_key: "64c3278f5a370776c95af3905ffd2bff"
server_key_iv: "7027d43730e9979f16fadf4617cd6b3a"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:05.508590 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "05fd30eadf2748c98f2692df82ec4b51"
format_stamp: "Formatted at 2026-08-12 14:25:05 on dist-test-slave-8w9v"
server_key: "64c3278f5a370776c95af3905ffd2bff"
server_key_iv: "7027d43730e9979f16fadf4617cd6b3a"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:05.510628 20178 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 14:25:05.524842 20197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:05.525082 20178 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:25:05.525193 20178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "05fd30eadf2748c98f2692df82ec4b51"
format_stamp: "Formatted at 2026-08-12 14:25:05 on dist-test-slave-8w9v"
server_key: "64c3278f5a370776c95af3905ffd2bff"
server_key_iv: "7027d43730e9979f16fadf4617cd6b3a"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:05.525298 20178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:05.537797 20178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:05.540412 20178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:05.540609 20178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:05.540894 20178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:05.541292 20178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:05.541375 20178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:05.541435 20178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:05.541479 20178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:05.548465 20178 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:34125
I20260812 14:25:05.548944 20178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:25:05.555142 20342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:34125 every 8 connection(s)
I20260812 14:25:05.557947 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 20178
I20260812 14:25:05.558065 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:25:05.558246 5712 external_mini_cluster.cc:1428] Setting key 4ee90da5701d2d5ce370d9ba75d701d5
Aug 12 14:25:05 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544705, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:05.574659 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:05.555890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:32793 (local address 127.5.148.1:58999)
0812 14:25:05.561841 (+ 5951us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:05.561911 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:05.562600 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:05.563107 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:05.563113 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:05.563394 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:05.563729 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:05.563736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:05.564842 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:05.564843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:05.566144 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:05.566147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:05.566754 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:05.566756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:05.566913 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:05.567565 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:05.567577 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:05.570194 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:05.572712 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:05.572721 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:05.572723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:05.573094 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:05.573350 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:05.573351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:05.573351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:05.573519 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:05.573865 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:05.573873 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:05.574068 (+ 195us) client_negotiation.cc:769] Sending connection context
0812 14:25:05.574196 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 14:25:05.574387 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5889,"thread_start_us":149,"threads_started":1}
I20260812 14:25:05.575311 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:05.555930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:58999 (local address 127.5.148.62:32793)
0812 14:25:05.562833 (+ 6903us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:05.562835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:05.562844 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:25:05.562871 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:05.562872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:05.562885 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:05.562962 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:05.563866 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:05.564708 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:05.566259 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:05.566566 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:05.566644 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:05.570397 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:05.570400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:05.570400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:05.570410 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:05.572597 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:05.573190 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:05.573192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:05.573192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:05.573267 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:05.573609 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:05.573610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:05.573611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:05.573762 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:05.573833 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:05.575098 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0812 14:25:05.575197 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6869,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
I20260812 14:25:05.576277 20343 heartbeater.cc:347] Connected to a master server at 127.5.148.62:32793
I20260812 14:25:05.576390 20343 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:05.576627 20343 heartbeater.cc:511] Master 127.5.148.62:32793 requested a full tablet report, sending...
I20260812 14:25:05.577303 20080 ts_manager.cc:194] Registered new tserver with Master: 05fd30eadf2748c98f2692df82ec4b51 (127.5.148.1:34125)
I20260812 14:25:05.578219 20080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:58999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:25:05.607134 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.148.20:53979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:32793
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:33153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:05.748497 20349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:05.748963 20349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:05.749087 20349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:05.751212 20349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:05.751291 20349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:05.751343 20349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
Aug 12 14:25:05 dist-test-slave-8w9v krb5kdc[19504](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786544705, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:05.754683 20349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:33153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.148.20:53979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:32793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:05.755020 20349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:05.755344 20349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:05.761639 20361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:05.761662 20363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:05.761639 20360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:05.762988 20349 server_base.cc:1034] running on GCE node
I20260812 14:25:05.763233 20349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:05.763525 20349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:05.764703 20349 hybrid_clock.cc:648] HybridClock initialized: now 1786544705764674 us; error 39 us; skew 500 ppm
Aug 12 14:25:05 dist-test-slave-8w9v krb5kdc[19504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544705, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:05.768338 20349 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:25:05.768977 20349 webserver.cc:460] Webserver started at http://127.5.148.2:44027/ using document root <none> and password file <none>
I20260812 14:25:05.769302 20349 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:05.769378 20349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:05.769517 20349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:25:05 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544705, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:25:05.792061 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17"
format_stamp: "Formatted at 2026-08-12 14:25:05 on dist-test-slave-8w9v"
server_key: "5ffeb8b199c9c186be182a926c4930c7"
server_key_iv: "e80265211242d290c1fedb3431af9664"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:05.792596 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17"
format_stamp: "Formatted at 2026-08-12 14:25:05 on dist-test-slave-8w9v"
server_key: "5ffeb8b199c9c186be182a926c4930c7"
server_key_iv: "e80265211242d290c1fedb3431af9664"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:05.794126 20349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:25:05.811218 20372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:05.811421 20349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:05.811510 20349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17"
format_stamp: "Formatted at 2026-08-12 14:25:05 on dist-test-slave-8w9v"
server_key: "5ffeb8b199c9c186be182a926c4930c7"
server_key_iv: "e80265211242d290c1fedb3431af9664"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:05.811610 20349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:05.832919 20349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:05.835594 20349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:05.835814 20349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:05.836251 20349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:05.836742 20349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:05.836838 20349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:05.836928 20349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:05.836988 20349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:05.847446 20349 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:35657
I20260812 14:25:05.847565 20506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:35657 every 8 connection(s)
I20260812 14:25:05.848167 20349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:25:05.851318 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 20349
I20260812 14:25:05.851424 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:25:05.851614 5712 external_mini_cluster.cc:1428] Setting key 75d4929bb3e3ebac943200b846631aed
Aug 12 14:25:05 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544705, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:05.861833 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:05.849368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:32793 (local address 127.5.148.2:38119)
0812 14:25:05.849601 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:05.849651 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:05.850581 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:05.851082 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:05.851086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:05.851551 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:05.851983 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:05.851990 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:05.853072 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:05.853073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:05.854425 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:05.854428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:05.855122 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:05.855123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:05.855295 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:05.856060 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:05.856071 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:05.858016 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:05.860043 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:05.860050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:05.860053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:05.860374 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:05.860641 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:05.860642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:05.860642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:05.860780 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:05.861101 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:05.861108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:05.861291 (+ 183us) client_negotiation.cc:769] Sending connection context
0812 14:25:05.861411 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 14:25:05.861578 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":92,"threads_started":1}
I20260812 14:25:05.862385 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:05.849412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:38119 (local address 127.5.148.62:32793)
0812 14:25:05.849470 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:05.849472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:05.849749 (+ 277us) server_negotiation.cc:407] Connection header received
0812 14:25:05.850749 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:05.850751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:05.850775 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:05.850819 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:05.852103 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:05.852948 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:05.854534 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:05.854917 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:05.854965 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:05.858126 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:05.858127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:05.858128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:05.858135 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:05.859948 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:05.860489 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:05.860490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:05.860490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:05.860565 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:05.860855 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:05.860856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:05.860856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:05.861004 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:05.861071 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:05.862192 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0812 14:25:05.862292 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:25:05.863196 20507 heartbeater.cc:347] Connected to a master server at 127.5.148.62:32793
I20260812 14:25:05.863301 20507 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:05.863497 20507 heartbeater.cc:511] Master 127.5.148.62:32793 requested a full tablet report, sending...
I20260812 14:25:05.863859 20080 ts_manager.cc:194] Registered new tserver with Master: ef3760f2bdfc4e7baf98d80eb50a5a17 (127.5.148.2:35657)
I20260812 14:25:05.864385 20080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:38119
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:25:05.905341 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.148.20:53979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:32793
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:33153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:06.051354 20512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:06.051587 20512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:06.051628 20512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:06.053017 20512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:06.053063 20512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:06.053090 20512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:25:06.054797 20512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:33153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.148.20:53979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:32793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:06.055063 20512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:06.055341 20512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:06.061345 20518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:06.061385 20524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:06.061661 20519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:06.062662 20512 server_base.cc:1034] running on GCE node
I20260812 14:25:06.062945 20512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:06.063231 20512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:06.064395 20512 hybrid_clock.cc:648] HybridClock initialized: now 1786544706064363 us; error 50 us; skew 500 ppm
Aug 12 14:25:06 dist-test-slave-8w9v krb5kdc[19504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:06.068457 20512 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:25:06.069033 20512 webserver.cc:460] Webserver started at http://127.5.148.3:36021/ using document root <none> and password file <none>
I20260812 14:25:06.069382 20512 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:06.069463 20512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:06.069610 20512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:25:06 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:25:06.084091 20512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "155a352d6dcf471c994ef2d0a51de14b"
format_stamp: "Formatted at 2026-08-12 14:25:06 on dist-test-slave-8w9v"
server_key: "5083b94a5c3386e71daca43683b9e99c"
server_key_iv: "e52922e35efd9073134b3c0e4a5b7a25"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:06.084717 20512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "155a352d6dcf471c994ef2d0a51de14b"
format_stamp: "Formatted at 2026-08-12 14:25:06 on dist-test-slave-8w9v"
server_key: "5083b94a5c3386e71daca43683b9e99c"
server_key_iv: "e52922e35efd9073134b3c0e4a5b7a25"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:06.086351 20512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:25:06.102384 20533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:06.102677 20512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260812 14:25:06.102787 20512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "155a352d6dcf471c994ef2d0a51de14b"
format_stamp: "Formatted at 2026-08-12 14:25:06 on dist-test-slave-8w9v"
server_key: "5083b94a5c3386e71daca43683b9e99c"
server_key_iv: "e52922e35efd9073134b3c0e4a5b7a25"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:06.102936 20512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:06.128811 20512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:06.131822 20512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:06.132020 20512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:06.132397 20512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:06.132946 20512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:06.133009 20512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:06.133049 20512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:06.133073 20512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:06.150621 20512 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:43701
I20260812 14:25:06.151142 20512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:25:06.152516 20669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:43701 every 8 connection(s)
I20260812 14:25:06.154955 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 20512
I20260812 14:25:06.155079 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:25:06.155282 5712 external_mini_cluster.cc:1428] Setting key 7aa993607619accd37868e1ca993c3b6
Aug 12 14:25:06 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:06.165122 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:06.153575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:50743 (local address 127.5.148.62:32793)
0812 14:25:06.153631 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:06.153634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:06.153879 (+ 245us) server_negotiation.cc:407] Connection header received
0812 14:25:06.154649 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:06.154650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:06.154664 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:06.154698 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:06.155630 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:06.156521 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:06.158165 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:06.158517 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:06.158591 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:06.161406 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:06.161408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:06.161409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:06.161418 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:06.163252 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:06.163873 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:06.163875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:06.163875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:06.163949 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:06.164314 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:06.164315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:06.164316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:06.164472 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:06.164556 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:06.164925 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 14:25:06.165026 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 14:25:06.165385 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:06.153228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:32793 (local address 127.5.148.3:50743)
0812 14:25:06.153736 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:06.153786 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:06.154509 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:06.154901 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:06.154906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:06.155197 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:06.155507 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:06.155514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:06.156658 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:06.156659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:06.158054 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:06.158058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:06.158624 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:06.158625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:06.158778 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:06.159528 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:06.159540 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:06.161291 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:06.163433 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:06.163441 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:06.163443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:06.163762 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:06.164047 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:06.164049 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:06.164049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:06.164226 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:06.164560 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:06.164568 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:06.164752 (+ 184us) client_negotiation.cc:769] Sending connection context
0812 14:25:06.164877 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 14:25:06.165067 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":121,"threads_started":1}
I20260812 14:25:06.166043 20670 heartbeater.cc:347] Connected to a master server at 127.5.148.62:32793
I20260812 14:25:06.166159 20670 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:06.166384 20670 heartbeater.cc:511] Master 127.5.148.62:32793 requested a full tablet report, sending...
I20260812 14:25:06.166852 20080 ts_manager.cc:194] Registered new tserver with Master: 155a352d6dcf471c994ef2d0a51de14b (127.5.148.3:43701)
I20260812 14:25:06.167088 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:25:06.167431 20080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:50743
I20260812 14:25:06.167824 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 20178
I20260812 14:25:06.191690 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 20349
I20260812 14:25:06.199263 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 20512
I20260812 14:25:06.207047 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 20038
I20260812 14:25:06.215543 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:32793
--webserver_interface=127.5.148.62
--webserver_port=37809
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:33153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:32793
--ranger_config_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.148.20:53979/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.148.20:53979/kms with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:06.299360 20673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:06.299621 20673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:06.299659 20673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:06.300849 20673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:25:06.300931 20673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:06.300954 20673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:06.300976 20673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:25:06.301003 20673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:25:06.302712 20673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:33153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.148.20:53979/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:32793
--ranger_config_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:32793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=37809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:06.303005 20673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:06.303244 20673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:06.308342 20684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:06.308398 20685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:06.308416 20687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:06.308593 20673 server_base.cc:1034] running on GCE node
I20260812 14:25:06.308845 20673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:06.309154 20673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:06.310312 20673 hybrid_clock.cc:648] HybridClock initialized: now 1786544706310275 us; error 48 us; skew 500 ppm
Aug 12 14:25:06 dist-test-slave-8w9v krb5kdc[19504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:06.313030 20673 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:25:06.313496 20673 webserver.cc:460] Webserver started at http://127.5.148.62:37809/ using document root <none> and password file <none>
I20260812 14:25:06.313747 20673 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:06.313832 20673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 14:25:06 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:25:06.399886 20673 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:06.400774 20698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:06.400974 20673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:25:06.401064 20673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "eb7dacc1c2d0496b9dab9293b74734ed"
format_stamp: "Formatted at 2026-08-12 14:25:04 on dist-test-slave-8w9v"
server_key: "a808f9ba9952815cea5415d1a4d29549"
server_key_iv: "b3b85a98f65eee891fb7f268dfaad945"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:06.401562 20673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:06.409523 20673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:06.411634 20673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:06.411818 20673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:06.416355 20673 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:32793
I20260812 14:25:06.416409 20763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:32793 every 8 connection(s)
I20260812 14:25:06.416813 20673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:25:06.418084 20764 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:25:06.418676 20764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed: Bootstrap starting.
I20260812 14:25:06.424623 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 20673
I20260812 14:25:06.424937 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.148.20:53979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:34125
--local_ip_for_outbound_sockets=127.5.148.1
--tserver_master_addrs=127.5.148.62:32793
--webserver_port=42021
--webserver_interface=127.5.148.1
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:33153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.148.20:53979/kms with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:25:06.427052 20764 log.cc:824] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:06.428929 20764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:25:06.429293 20764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed: Bootstrap complete.
I20260812 14:25:06.431988 20764 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32793 } }
I20260812 14:25:06.432395 20764 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7dacc1c2d0496b9dab9293b74734ed, State: Initialized, Role: FOLLOWER
I20260812 14:25:06.432633 20764 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32793 } }
I20260812 14:25:06.432740 20764 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:25:06.432821 20764 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:25:06.432876 20764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:25:06.434015 20764 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32793 } }
I20260812 14:25:06.434105 20764 leader_election.cc:302] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb7dacc1c2d0496b9dab9293b74734ed; no voters:
I20260812 14:25:06.434299 20764 leader_election.cc:288] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 14:25:06.434372 20768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 2 FOLLOWER]: Leader election won for term 2
I20260812 14:25:06.434530 20768 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [term 2 LEADER]: Becoming Leader. State: Replica: eb7dacc1c2d0496b9dab9293b74734ed, State: Running, Role: LEADER
I20260812 14:25:06.434589 20764 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:25:06.434743 20768 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32793 } }
I20260812 14:25:06.435081 20768 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7dacc1c2d0496b9dab9293b74734ed. Latest consensus state: current_term: 2 leader_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32793 } } }
I20260812 14:25:06.435166 20768 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:06.435328 20769 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7dacc1c2d0496b9dab9293b74734ed" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32793 } } }
I20260812 14:25:06.435405 20769 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:06.435454 20774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:25:06.436094 20774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:25:06.436334 20774 catalog_manager.cc:1259] Loaded cluster ID: 5876acaaf5cf460197d17ddfeaf6335d
I20260812 14:25:06.436425 20774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:25:06.436620 20764 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 14:25:06.437786 20774 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:25:06.437985 20774 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed: Loaded TSK: 0
I20260812 14:25:06.438196 20774 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:25:06.519703 20766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:06.519938 20766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:06.520004 20766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:06.521544 20766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:06.521642 20766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:06.521701 20766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:25:06.523280 20766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:33153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.148.20:53979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:34125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=42021
--tserver_master_addrs=127.5.148.62:32793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:06.523554 20766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:06.523816 20766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:06.529692 20811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:06.529897 20766 server_base.cc:1034] running on GCE node
W20260812 14:25:06.529703 20808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:06.529987 20809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:06.530329 20766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:06.530622 20766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:06.533890 20766 hybrid_clock.cc:648] HybridClock initialized: now 1786544706533858 us; error 39 us; skew 500 ppm
Aug 12 14:25:06 dist-test-slave-8w9v krb5kdc[19504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:06.536844 20766 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:25:06.537299 20766 webserver.cc:460] Webserver started at http://127.5.148.1:42021/ using document root <none> and password file <none>
I20260812 14:25:06.537576 20766 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:06.537667 20766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 14:25:06 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:25:06.562651 20766 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260812 14:25:06.563974 20822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:06.564236 20766 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:25:06.564334 20766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "05fd30eadf2748c98f2692df82ec4b51"
format_stamp: "Formatted at 2026-08-12 14:25:05 on dist-test-slave-8w9v"
server_key: "64c3278f5a370776c95af3905ffd2bff"
server_key_iv: "7027d43730e9979f16fadf4617cd6b3a"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:06.564819 20766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:06.584740 20766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:06.587432 20766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:06.587664 20766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:06.588014 20766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:06.588495 20766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:06.588575 20766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:06.588639 20766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:06.588708 20766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:06.596220 20766 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:34125
I20260812 14:25:06.596362 20961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:34125 every 8 connection(s)
I20260812 14:25:06.596730 20766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:25:06.602239 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 20766
I20260812 14:25:06.602612 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.148.20:53979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:35657
--local_ip_for_outbound_sockets=127.5.148.2
--tserver_master_addrs=127.5.148.62:32793
--webserver_port=44027
--webserver_interface=127.5.148.2
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:33153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.148.20:53979/kms with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:25:06 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:06.611706 20965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:06.597530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:32793 (local address 127.5.148.1:46341)
0812 14:25:06.597872 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:06.597918 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:06.598579 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:06.599139 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:06.599143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:06.599398 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:06.600623 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:06.600631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:06.602450 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:06.602452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:06.603735 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:06.603739 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:06.604798 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:06.604799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:06.605005 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:06.605629 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:06.605639 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:06.607331 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:06.609678 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:06.609686 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:06.609688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:06.610000 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:06.610309 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:06.610310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:06.610310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:06.610447 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:06.610995 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:06.611003 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:06.611164 (+ 161us) client_negotiation.cc:769] Sending connection context
0812 14:25:06.611282 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 14:25:06.611464 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":89,"threads_started":1}
I20260812 14:25:06.612437 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:06.597632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:46341 (local address 127.5.148.62:32793)
0812 14:25:06.597874 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:06.597883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:06.598060 (+ 177us) server_negotiation.cc:407] Connection header received
0812 14:25:06.598851 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:06.598858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:06.598926 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:06.599069 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:06.600769 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:06.602309 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:06.603841 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:06.604545 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:06.604672 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:06.607456 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:06.607461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:06.607461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:06.607474 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:06.609579 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:06.610106 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:06.610110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:06.610111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:06.610226 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:06.610530 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:06.610532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:06.610532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:06.610855 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:06.610961 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:06.612038 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0812 14:25:06.612202 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":105,"threads_started":1}
I20260812 14:25:06.613332 20963 heartbeater.cc:347] Connected to a master server at 127.5.148.62:32793
I20260812 14:25:06.613425 20963 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:06.613608 20963 heartbeater.cc:511] Master 127.5.148.62:32793 requested a full tablet report, sending...
W20260812 14:25:06.614468 20963 heartbeater.cc:644] Failed to heartbeat to 127.5.148.62:32793 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 14:25:06.614531 20963 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:06.614586 20963 heartbeater.cc:511] Master 127.5.148.62:32793 requested a full tablet report, sending...
I20260812 14:25:06.614969 20963 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:06.615037 20963 heartbeater.cc:511] Master 127.5.148.62:32793 requested a full tablet report, sending...
W20260812 14:25:06.615357 20963 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:25:06.720592 20967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:06.720978 20967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:06.721067 20967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:06.722541 20967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:06.722630 20967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:06.722675 20967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:25:06.724469 20967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:33153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.148.20:53979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:35657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=44027
--tserver_master_addrs=127.5.148.62:32793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:06.724771 20967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:06.725041 20967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:06.731616 20974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:06.731835 20977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:06.731616 20973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:06.733127 20967 server_base.cc:1034] running on GCE node
I20260812 14:25:06.733428 20967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:06.733757 20967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:06.737134 20967 hybrid_clock.cc:648] HybridClock initialized: now 1786544706737105 us; error 42 us; skew 500 ppm
Aug 12 14:25:06 dist-test-slave-8w9v krb5kdc[19504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:06.740669 20967 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:25:06.741343 20967 webserver.cc:460] Webserver started at http://127.5.148.2:44027/ using document root <none> and password file <none>
I20260812 14:25:06.741657 20967 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:06.741834 20967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 14:25:06 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:25:06.761152 20967 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:06.761991 20987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:06.762202 20967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:06.762338 20967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17"
format_stamp: "Formatted at 2026-08-12 14:25:05 on dist-test-slave-8w9v"
server_key: "5ffeb8b199c9c186be182a926c4930c7"
server_key_iv: "e80265211242d290c1fedb3431af9664"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:06.762884 20967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:06.780790 20967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:06.784227 20967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:06.784432 20967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:06.784749 20967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:06.785183 20967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:06.785239 20967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:06.785274 20967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:06.785305 20967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:06.793365 20967 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:35657
I20260812 14:25:06.793922 20967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:25:06.794757 21125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:35657 every 8 connection(s)
I20260812 14:25:06.802491 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 20967
I20260812 14:25:06.802822 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.148.20:53979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:43701
--local_ip_for_outbound_sockets=127.5.148.3
--tserver_master_addrs=127.5.148.62:32793
--webserver_port=36021
--webserver_interface=127.5.148.3
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:33153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.148.20:53979/kms with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:25:06 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:06.808096 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:06.796733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:32793 (local address 127.5.148.2:41423)
0812 14:25:06.797016 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:06.797060 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:06.797724 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:06.798070 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:06.798075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:06.798351 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:06.798639 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:06.798646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:06.799778 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:06.799779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:06.801069 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:06.801073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:06.801631 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:06.801632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:06.801792 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:06.802638 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:06.802649 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:06.804240 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:06.806334 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:06.806343 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:06.806346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:06.806672 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:06.806898 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:06.806899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:06.806900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:06.807045 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:06.807340 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:06.807346 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:06.807528 (+ 182us) client_negotiation.cc:769] Sending connection context
0812 14:25:06.807658 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 14:25:06.807830 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":93,"threads_started":1}
I20260812 14:25:06.808570 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:06.796750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:41423 (local address 127.5.148.62:32793)
0812 14:25:06.796833 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:06.796836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:06.797171 (+ 335us) server_negotiation.cc:407] Connection header received
0812 14:25:06.797866 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:06.797868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:06.797882 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:06.797972 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:06.798781 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:06.799654 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:06.801187 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:06.801480 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:06.801576 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:06.804379 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:06.804381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:06.804382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:06.804391 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:06.806234 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:06.806762 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:06.806764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:06.806764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:06.806831 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:06.807117 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:06.807118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:06.807118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:06.807257 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:06.807315 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:06.808381 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0812 14:25:06.808478 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 14:25:06.808939 21126 heartbeater.cc:347] Connected to a master server at 127.5.148.62:32793
I20260812 14:25:06.809106 21126 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:06.809388 21126 heartbeater.cc:511] Master 127.5.148.62:32793 requested a full tablet report, sending...
W20260812 14:25:06.812399 21126 heartbeater.cc:644] Failed to heartbeat to 127.5.148.62:32793 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 14:25:06.812508 21126 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:06.812594 21126 heartbeater.cc:511] Master 127.5.148.62:32793 requested a full tablet report, sending...
I20260812 14:25:06.812942 21126 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:06.813032 21126 heartbeater.cc:511] Master 127.5.148.62:32793 requested a full tablet report, sending...
W20260812 14:25:06.813436 21126 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:25:06.940904 21129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:06.941303 21129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:06.941409 21129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:06.943416 21129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:06.943518 21129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:06.943583 21129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:25:06.946146 21129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:33153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.148.20:53979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:43701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=36021
--tserver_master_addrs=127.5.148.62:32793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:06.946444 21129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:06.946818 21129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:06.953596 21139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:06.953920 21141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:06.954061 21137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:06.954533 21129 server_base.cc:1034] running on GCE node
I20260812 14:25:06.954764 21129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:06.955070 21129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:06.956207 21129 hybrid_clock.cc:648] HybridClock initialized: now 1786544706956134 us; error 88 us; skew 500 ppm
Aug 12 14:25:06 dist-test-slave-8w9v krb5kdc[19504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:06.959867 21129 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:25:06.960448 21129 webserver.cc:460] Webserver started at http://127.5.148.3:36021/ using document root <none> and password file <none>
I20260812 14:25:06.960740 21129 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:06.960827 21129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 14:25:06 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:25:06.991537 21129 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260812 14:25:06.992756 21151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:06.992954 21129 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:06.993052 21129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "155a352d6dcf471c994ef2d0a51de14b"
format_stamp: "Formatted at 2026-08-12 14:25:06 on dist-test-slave-8w9v"
server_key: "5083b94a5c3386e71daca43683b9e99c"
server_key_iv: "e52922e35efd9073134b3c0e4a5b7a25"
server_key_version: "kuduclusterkey@0"
I20260812 14:25:06.993566 21129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:07.010326 21129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:07.013214 21129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:07.013490 21129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:07.013827 21129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:07.014297 21129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:07.014408 21129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:07.014492 21129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:07.014561 21129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:07.022492 21129 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:43701
I20260812 14:25:07.023154 21129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:25:07.024509 21283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:43701 every 8 connection(s)
I20260812 14:25:07.027375 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 21129
Aug 12 14:25:07 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:07.038466 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:07.028072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.5.148.62:32793)
0812 14:25:07.028464 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:07.028466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:07.028474 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:25:07.028497 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:07.028498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:07.028511 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:07.028732 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:07.029657 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:07.030363 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:07.031542 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:07.031754 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:07.031848 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:07.033521 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:07.033522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:07.033523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:07.033532 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:07.036446 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:07.037004 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:07.037006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:07.037006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:07.037081 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:07.037320 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:07.037321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:07.037321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:07.037475 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:07.037660 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:07.038227 (+ 567us) server_negotiation.cc:299] Negotiation successful
0812 14:25:07.038343 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":101,"threads_started":1}
I20260812 14:25:07.038695 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:07.025292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:45725 (local address 127.5.148.62:32793)
0812 14:25:07.025353 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:07.025356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:07.025611 (+ 255us) server_negotiation.cc:407] Connection header received
0812 14:25:07.026237 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:07.026238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:07.026255 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:07.026322 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:07.026978 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:07.027718 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:07.029483 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:07.029814 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:07.029863 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:07.034150 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:07.034152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:07.034152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:07.034161 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:07.036215 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:07.037491 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:07.037493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:07.037494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:07.037585 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:07.037938 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:07.037939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:07.037939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:07.038072 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:07.038114 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:07.038525 (+ 411us) server_negotiation.cc:299] Negotiation successful
0812 14:25:07.038608 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":29}
I20260812 14:25:07.039209 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:07.025304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:32793 (local address 127.5.148.3:45725)
0812 14:25:07.025470 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:07.025515 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:07.026094 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:07.026406 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:07.026411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:07.026632 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:07.026858 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:07.026863 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:07.027870 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:07.027884 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:07.029356 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:07.029359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:07.029990 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:07.029991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:07.030138 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:07.030826 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:07.030838 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:07.033976 (+ 3138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:07.036987 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:07.036997 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:07.037001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:07.037388 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:07.037670 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:07.037671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:07.037671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:07.037850 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:07.038164 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:07.038172 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:07.038358 (+ 186us) client_negotiation.cc:769] Sending connection context
0812 14:25:07.038754 (+ 396us) client_negotiation.cc:240] Negotiation successful
0812 14:25:07.038950 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260812 14:25:07.040972 21284 heartbeater.cc:347] Connected to a master server at 127.5.148.62:32793
I20260812 14:25:07.041085 21284 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:07.041288 21284 heartbeater.cc:511] Master 127.5.148.62:32793 requested a full tablet report, sending...
W20260812 14:25:07.042014 21284 heartbeater.cc:644] Failed to heartbeat to 127.5.148.62:32793 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 14:25:07.042085 21284 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:07.042145 21284 heartbeater.cc:511] Master 127.5.148.62:32793 requested a full tablet report, sending...
I20260812 14:25:07.042490 21284 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:07.042560 21284 heartbeater.cc:511] Master 127.5.148.62:32793 requested a full tablet report, sending...
W20260812 14:25:07.042984 21284 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 14:25:07.615536 20963 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:07.615738 20963 heartbeater.cc:511] Master 127.5.148.62:32793 requested a full tablet report, sending...
I20260812 14:25:07.616214 20717 ts_manager.cc:194] Registered new tserver with Master: 05fd30eadf2748c98f2692df82ec4b51 (127.5.148.1:34125)
I20260812 14:25:07.617261 20717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:46341
Aug 12 14:25:07 dist-test-slave-8w9v krb5kdc[19504](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786544707, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:07.813639 21126 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:07.813823 21126 heartbeater.cc:511] Master 127.5.148.62:32793 requested a full tablet report, sending...
I20260812 14:25:07.814184 20717 ts_manager.cc:194] Registered new tserver with Master: ef3760f2bdfc4e7baf98d80eb50a5a17 (127.5.148.2:35657)
I20260812 14:25:07.814733 20717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:41423
I20260812 14:25:08.043256 21284 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:08.043427 21284 heartbeater.cc:511] Master 127.5.148.62:32793 requested a full tablet report, sending...
I20260812 14:25:08.043803 20717 ts_manager.cc:194] Registered new tserver with Master: 155a352d6dcf471c994ef2d0a51de14b (127.5.148.3:43701)
I20260812 14:25:08.044394 20717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:45725
I20260812 14:25:08.048406 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:25:08.065184 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:08.051850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.5.148.62:32793)
0812 14:25:08.055602 (+ 3752us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:08.055604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:08.055613 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:25:08.055643 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:08.055644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:08.055657 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:08.056536 (+ 879us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:08.056578 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.057396 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.058517 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.058826 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.059203 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:08.060596 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:08.060597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:08.060597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:08.060615 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:08.062857 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:08.063427 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:08.063428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:08.063429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:08.063495 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:08.063703 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:08.063704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:08.063704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:08.063842 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:08.064930 (+ 1088us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:08.064977 (+ 47us) server_negotiation.cc:299] Negotiation successful
0812 14:25:08.065070 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3728,"thread_start_us":85,"threads_started":1}
I20260812 14:25:08.068809 20717 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:25:08.071376 20717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 12 14:25:08 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544707, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:25:08.100680 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:08.089233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:34125 (local address 127.0.0.1:54682)
0812 14:25:08.092543 (+ 3310us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:08.092845 (+ 302us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:08.092904 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:08.096529 (+ 3625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:08.096533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:08.096536 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:08.096632 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:08.096633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.097967 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.097968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:08.099264 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:08.099267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.100355 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.100356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:08.100445 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:08.100459 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 14:25:08.100530 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 14:25:08.100588 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3286,"thread_start_us":137,"threads_started":1}
I20260812 14:25:08.101230 21319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:08.089608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54682 (local address 127.5.148.1:34125)
0812 14:25:08.091954 (+ 2346us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:08.091961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:08.092992 (+ 1031us) server_negotiation.cc:407] Connection header received
0812 14:25:08.093039 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:08.093042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:08.093061 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:08.093139 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:08.096739 (+ 3600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.097864 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.099369 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.100167 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.100236 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:08.100303 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:08.101103 (+ 800us) server_negotiation.cc:299] Negotiation successful
0812 14:25:08.101151 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2304,"thread_start_us":84,"threads_started":1}
I20260812 14:25:08.102272 20882 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae79260e59a478ba024103d6d635cba (DEFAULT_TABLE table=test-table [id=6718698e629a4dd999f2b94ff67f2544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:25:08.102679 20882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae79260e59a478ba024103d6d635cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:08.103328 21316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:08.089230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:43701 (local address 127.0.0.1:37252)
0812 14:25:08.092166 (+ 2936us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:08.092225 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:08.092308 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:08.097589 (+ 5281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:08.097590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:08.097592 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:08.097651 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:08.097653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.100578 (+ 2925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.100579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:08.101724 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:08.101727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.102759 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.102761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:08.102835 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:08.102840 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:25:08.102868 (+ 28us) client_negotiation.cc:240] Negotiation successful
0812 14:25:08.102906 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2877,"thread_start_us":124,"threads_started":1}
I20260812 14:25:08.103754 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:08.089835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.5.148.3:43701)
0812 14:25:08.097337 (+ 7502us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:08.097342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:08.097350 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:25:08.097380 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:08.097381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:08.097394 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:08.097488 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:08.098169 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.100478 (+ 2309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.101897 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.102659 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.103528 (+ 869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:08.103601 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:08.103628 (+ 27us) server_negotiation.cc:299] Negotiation successful
0812 14:25:08.103672 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7467,"thread_start_us":66,"threads_started":1}
I20260812 14:25:08.104260 21317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:08.089542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:35657 (local address 127.0.0.1:33786)
0812 14:25:08.097179 (+ 7637us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:08.097189 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:08.097234 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:08.098033 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:08.098035 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:08.098036 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:08.098080 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:08.098082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.101848 (+ 3766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.101849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:08.102922 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:08.102924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.103991 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.103992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:08.104060 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:08.104064 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:25:08.104114 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:25:08.104193 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7619,"thread_start_us":66,"threads_started":1}
I20260812 14:25:08.104519 21321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:08.089863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.5.148.2:35657)
0812 14:25:08.097786 (+ 7923us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:08.097792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:08.097802 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:25:08.097835 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:08.097838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:08.097853 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:08.097934 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:08.099081 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.100075 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.103013 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.103828 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.103900 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:08.103954 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:08.104370 (+ 416us) server_negotiation.cc:299] Negotiation successful
0812 14:25:08.104421 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7881,"thread_start_us":68,"threads_started":1}
I20260812 14:25:08.104472 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae79260e59a478ba024103d6d635cba (DEFAULT_TABLE table=test-table [id=6718698e629a4dd999f2b94ff67f2544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:25:08.104856 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae79260e59a478ba024103d6d635cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:08.105176 21045 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae79260e59a478ba024103d6d635cba (DEFAULT_TABLE table=test-table [id=6718698e629a4dd999f2b94ff67f2544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:25:08.105511 21045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae79260e59a478ba024103d6d635cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:08.108035 21323 tablet_bootstrap.cc:492] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b: Bootstrap starting.
I20260812 14:25:08.109023 21323 tablet_bootstrap.cc:654] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:08.109347 21323 log.cc:824] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:08.110060 21326 tablet_bootstrap.cc:492] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17: Bootstrap starting.
I20260812 14:25:08.110773 21326 tablet_bootstrap.cc:654] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:08.111227 21326 log.cc:824] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:08.112876 21328 tablet_bootstrap.cc:492] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51: Bootstrap starting.
I20260812 14:25:08.113756 21328 tablet_bootstrap.cc:654] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:08.114315 21328 log.cc:824] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:08.115605 21326 tablet_bootstrap.cc:492] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17: No bootstrap required, opened a new log
I20260812 14:25:08.115727 21326 ts_tablet_manager.cc:1397] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 14:25:08.118654 21326 raft_consensus.cc:348] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155a352d6dcf471c994ef2d0a51de14b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 43701 } } peers { permanent_uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35657 } } peers { permanent_uuid: "05fd30eadf2748c98f2692df82ec4b51" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 34125 } }
I20260812 14:25:08.119891 21328 tablet_bootstrap.cc:492] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51: No bootstrap required, opened a new log
I20260812 14:25:08.120024 21328 ts_tablet_manager.cc:1397] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260812 14:25:08.120234 21326 raft_consensus.cc:374] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:08.120349 21326 raft_consensus.cc:729] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3760f2bdfc4e7baf98d80eb50a5a17, State: Initialized, Role: FOLLOWER
I20260812 14:25:08.120522 21326 consensus_queue.cc:260] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155a352d6dcf471c994ef2d0a51de14b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 43701 } } peers { permanent_uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35657 } } peers { permanent_uuid: "05fd30eadf2748c98f2692df82ec4b51" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 34125 } }
I20260812 14:25:08.120945 21326 ts_tablet_manager.cc:1428] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260812 14:25:08.121135 21126 heartbeater.cc:503] Master 127.5.148.62:32793 was elected leader, sending a full tablet report...
I20260812 14:25:08.123303 21328 raft_consensus.cc:348] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155a352d6dcf471c994ef2d0a51de14b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 43701 } } peers { permanent_uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35657 } } peers { permanent_uuid: "05fd30eadf2748c98f2692df82ec4b51" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 34125 } }
I20260812 14:25:08.123551 21328 raft_consensus.cc:374] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:08.123641 21328 raft_consensus.cc:729] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fd30eadf2748c98f2692df82ec4b51, State: Initialized, Role: FOLLOWER
I20260812 14:25:08.123849 21328 consensus_queue.cc:260] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155a352d6dcf471c994ef2d0a51de14b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 43701 } } peers { permanent_uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35657 } } peers { permanent_uuid: "05fd30eadf2748c98f2692df82ec4b51" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 34125 } }
I20260812 14:25:08.124217 21328 ts_tablet_manager.cc:1428] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260812 14:25:08.124384 20963 heartbeater.cc:503] Master 127.5.148.62:32793 was elected leader, sending a full tablet report...
I20260812 14:25:08.124897 21323 tablet_bootstrap.cc:492] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b: No bootstrap required, opened a new log
I20260812 14:25:08.125046 21323 ts_tablet_manager.cc:1397] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260812 14:25:08.127903 21323 raft_consensus.cc:348] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155a352d6dcf471c994ef2d0a51de14b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 43701 } } peers { permanent_uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35657 } } peers { permanent_uuid: "05fd30eadf2748c98f2692df82ec4b51" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 34125 } }
I20260812 14:25:08.128985 21323 raft_consensus.cc:374] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:08.129122 21323 raft_consensus.cc:729] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155a352d6dcf471c994ef2d0a51de14b, State: Initialized, Role: FOLLOWER
I20260812 14:25:08.129383 21323 consensus_queue.cc:260] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155a352d6dcf471c994ef2d0a51de14b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 43701 } } peers { permanent_uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35657 } } peers { permanent_uuid: "05fd30eadf2748c98f2692df82ec4b51" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 34125 } }
I20260812 14:25:08.129752 21323 ts_tablet_manager.cc:1428] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260812 14:25:08.129909 21284 heartbeater.cc:503] Master 127.5.148.62:32793 was elected leader, sending a full tablet report...
W20260812 14:25:08.276598 21285 tablet.cc:2369] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:25:08.295686 21127 tablet.cc:2369] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:25:08.348446 20964 tablet.cc:2369] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 14:25:08 dist-test-slave-8w9v krb5kdc[19504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544707, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for HTTP/127.5.148.20@KRBTEST.COM
I20260812 14:25:08.433802 21331 raft_consensus.cc:482] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:25:08.433977 21331 raft_consensus.cc:504] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155a352d6dcf471c994ef2d0a51de14b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 43701 } } peers { permanent_uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35657 } } peers { permanent_uuid: "05fd30eadf2748c98f2692df82ec4b51" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 34125 } }
I20260812 14:25:08.434419 21331 leader_election.cc:288] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155a352d6dcf471c994ef2d0a51de14b (127.5.148.3:43701), ef3760f2bdfc4e7baf98d80eb50a5a17 (127.5.148.2:35657)
I20260812 14:25:08.438594 21321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:08.434874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:37153 (local address 127.5.148.2:35657)
0812 14:25:08.434950 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:08.434951 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:08.435063 (+ 112us) server_negotiation.cc:407] Connection header received
0812 14:25:08.435180 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:08.435182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:08.435196 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:08.435241 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:08.435417 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.436153 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.437590 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.438171 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.438224 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:08.438235 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:08.438475 (+ 240us) server_negotiation.cc:299] Negotiation successful
0812 14:25:08.438517 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 14:25:08.439136 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:08.434582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:35657 (local address 127.5.148.1:37153)
0812 14:25:08.435003 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:08.435015 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:08.435086 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:08.435285 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:08.435286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:08.435288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:08.435340 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:08.435341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.436271 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.436272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:08.437468 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:08.437473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.438314 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.438315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:08.438391 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:08.438395 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:25:08.438963 (+ 568us) client_negotiation.cc:240] Negotiation successful
0812 14:25:08.439014 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":3,"thread_start_us":103,"threads_started":1}
I20260812 14:25:08.439692 21068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae79260e59a478ba024103d6d635cba" candidate_uuid: "05fd30eadf2748c98f2692df82ec4b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17" is_pre_election: true
I20260812 14:25:08.439895 21068 raft_consensus.cc:2454] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05fd30eadf2748c98f2692df82ec4b51 in term 0.
I20260812 14:25:08.440300 20825 leader_election.cc:302] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fd30eadf2748c98f2692df82ec4b51, ef3760f2bdfc4e7baf98d80eb50a5a17; no voters:
I20260812 14:25:08.440526 21331 raft_consensus.cc:2781] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:25:08.440590 21331 raft_consensus.cc:482] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:25:08.440614 21331 raft_consensus.cc:3037] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:08.441780 21331 raft_consensus.cc:504] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155a352d6dcf471c994ef2d0a51de14b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 43701 } } peers { permanent_uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35657 } } peers { permanent_uuid: "05fd30eadf2748c98f2692df82ec4b51" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 34125 } }
I20260812 14:25:08.441944 21331 leader_election.cc:288] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [CANDIDATE]: Term 1 election: Requested vote from peers 155a352d6dcf471c994ef2d0a51de14b (127.5.148.3:43701), ef3760f2bdfc4e7baf98d80eb50a5a17 (127.5.148.2:35657)
I20260812 14:25:08.442281 21068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae79260e59a478ba024103d6d635cba" candidate_uuid: "05fd30eadf2748c98f2692df82ec4b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17"
I20260812 14:25:08.442368 21068 raft_consensus.cc:3037] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:08.443356 21068 raft_consensus.cc:2454] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05fd30eadf2748c98f2692df82ec4b51 in term 1.
I20260812 14:25:08.443611 20825 leader_election.cc:302] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fd30eadf2748c98f2692df82ec4b51, ef3760f2bdfc4e7baf98d80eb50a5a17; no voters:
I20260812 14:25:08.443758 21331 raft_consensus.cc:2781] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:25:08.443946 21331 raft_consensus.cc:686] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [term 1 LEADER]: Becoming Leader. State: Replica: 05fd30eadf2748c98f2692df82ec4b51, State: Running, Role: LEADER
I20260812 14:25:08.443980 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:08.434808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:45541 (local address 127.5.148.3:43701)
0812 14:25:08.434887 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:08.434890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:08.434898 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:25:08.434930 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:08.434931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:08.434942 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:08.435009 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:08.435228 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.436060 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.442920 (+ 6860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.443538 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.443585 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:08.443599 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:08.443846 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 14:25:08.443905 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:25:08.444053 21331 consensus_queue.cc:237] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155a352d6dcf471c994ef2d0a51de14b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 43701 } } peers { permanent_uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35657 } } peers { permanent_uuid: "05fd30eadf2748c98f2692df82ec4b51" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 34125 } }
I20260812 14:25:08.444231 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:08.434582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:43701 (local address 127.5.148.1:45541)
0812 14:25:08.434769 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:08.434781 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:08.434841 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:08.435032 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:08.435033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:08.435036 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:08.435130 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:08.435132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.441615 (+ 6483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.441616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:08.442791 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:08.442795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.443659 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.443668 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:08.443748 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:08.443753 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:25:08.444093 (+ 340us) client_negotiation.cc:240] Negotiation successful
0812 14:25:08.444162 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":105,"threads_started":1}
I20260812 14:25:08.444747 21230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae79260e59a478ba024103d6d635cba" candidate_uuid: "05fd30eadf2748c98f2692df82ec4b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155a352d6dcf471c994ef2d0a51de14b" is_pre_election: true
I20260812 14:25:08.445050 21230 raft_consensus.cc:2454] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05fd30eadf2748c98f2692df82ec4b51 in term 0.
I20260812 14:25:08.444751 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae79260e59a478ba024103d6d635cba" candidate_uuid: "05fd30eadf2748c98f2692df82ec4b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155a352d6dcf471c994ef2d0a51de14b"
I20260812 14:25:08.445220 21228 raft_consensus.cc:3037] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:08.445756 20717 catalog_manager.cc:5495] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05fd30eadf2748c98f2692df82ec4b51 (127.5.148.1). New cstate: current_term: 1 leader_uuid: "05fd30eadf2748c98f2692df82ec4b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155a352d6dcf471c994ef2d0a51de14b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05fd30eadf2748c98f2692df82ec4b51" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 34125 } health_report { overall_health: HEALTHY } } }
I20260812 14:25:08.446425 21228 raft_consensus.cc:2454] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05fd30eadf2748c98f2692df82ec4b51 in term 1.
I20260812 14:25:08.536260 21319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:08.532680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.5.148.1:34125)
0812 14:25:08.532744 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:08.532747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:08.532760 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:25:08.532822 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:08.532823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:08.532848 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:08.532925 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:25:08.533131 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.533899 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.535026 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:08.535339 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:08.535405 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:08.535621 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:25:08.535766 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:25:08.535854 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:08.536146 (+ 292us) server_negotiation.cc:299] Negotiation successful
0812 14:25:08.536169 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:25:08.538708 21068 raft_consensus.cc:1261] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17 [term 1 FOLLOWER]: Refusing update from remote peer 05fd30eadf2748c98f2692df82ec4b51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:25:08.539180 21337 consensus_queue.cc:1035] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3760f2bdfc4e7baf98d80eb50a5a17" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:25:08.539779 21228 raft_consensus.cc:1261] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b [term 1 FOLLOWER]: Refusing update from remote peer 05fd30eadf2748c98f2692df82ec4b51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:25:08.540042 21331 consensus_queue.cc:1035] T 9ae79260e59a478ba024103d6d635cba P 05fd30eadf2748c98f2692df82ec4b51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155a352d6dcf471c994ef2d0a51de14b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:25:08.542738 21340 mvcc.cc:205] Tried to move back new op lower bound from 7317687126171566080 to 7317687125787516928. Current Snapshot: MvccSnapshot[applied={T|T < 7317687126171566080}]
I20260812 14:25:08.543602 21341 mvcc.cc:205] Tried to move back new op lower bound from 7317687126171566080 to 7317687125787516928. Current Snapshot: MvccSnapshot[applied={T|T < 7317687126171566080}]
I20260812 14:25:08.545100 21344 mvcc.cc:205] Tried to move back new op lower bound from 7317687126171566080 to 7317687125787516928. Current Snapshot: MvccSnapshot[applied={T|T < 7317687126171566080}]
I20260812 14:25:08.545682 20717 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:25:08.545778 20717 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:25:08.546813 20717 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb7dacc1c2d0496b9dab9293b74734ed: Sending DeleteTablet for 3 replicas of tablet 9ae79260e59a478ba024103d6d635cba
I20260812 14:25:08.547173 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ae79260e59a478ba024103d6d635cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:08 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:37252
I20260812 14:25:08.547185 21045 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ae79260e59a478ba024103d6d635cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:08 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:33786
I20260812 14:25:08.547549 20882 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ae79260e59a478ba024103d6d635cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:08 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:54682
I20260812 14:25:08.548225 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 20766
I20260812 14:25:08.552229 21351 tablet_replica.cc:330] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17: stopping tablet replica
I20260812 14:25:08.552378 21351 raft_consensus.cc:2229] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:25:08.552508 21351 raft_consensus.cc:2258] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:25:08.553608 21351 ts_tablet_manager.cc:1905] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:25:08.560346 21351 ts_tablet_manager.cc:1918] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:25:08.560499 21350 tablet_replica.cc:330] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b: stopping tablet replica
I20260812 14:25:08.560527 21351 log.cc:1196] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/9ae79260e59a478ba024103d6d635cba
I20260812 14:25:08.560779 21350 raft_consensus.cc:2229] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:25:08.560833 21351 ts_tablet_manager.cc:1939] T 9ae79260e59a478ba024103d6d635cba P ef3760f2bdfc4e7baf98d80eb50a5a17: Deleting consensus metadata
I20260812 14:25:08.561028 21350 raft_consensus.cc:2258] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:25:08.561463 20701 catalog_manager.cc:4841] TS ef3760f2bdfc4e7baf98d80eb50a5a17 (127.5.148.2:35657): tablet 9ae79260e59a478ba024103d6d635cba (table test-table [id=6718698e629a4dd999f2b94ff67f2544]) successfully deleted
I20260812 14:25:08.564659 21350 ts_tablet_manager.cc:1905] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:25:08.566741 21350 ts_tablet_manager.cc:1918] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:25:08.566901 21350 log.cc:1196] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/9ae79260e59a478ba024103d6d635cba
I20260812 14:25:08.567263 21350 ts_tablet_manager.cc:1939] T 9ae79260e59a478ba024103d6d635cba P 155a352d6dcf471c994ef2d0a51de14b: Deleting consensus metadata
W20260812 14:25:08.567865 20699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:34125 (error 108)
I20260812 14:25:08.568208 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 20967
I20260812 14:25:08.571382 20702 catalog_manager.cc:4841] TS 155a352d6dcf471c994ef2d0a51de14b (127.5.148.3:43701): tablet 9ae79260e59a478ba024103d6d635cba (table test-table [id=6718698e629a4dd999f2b94ff67f2544]) successfully deleted
I20260812 14:25:08.580935 21317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:08.580745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:34125 (local address 127.0.0.1:54704)
0812 14:25:08.580815 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:08.580872 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:34125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 14:25:08.583914 20699 catalog_manager.cc:4584] TS 05fd30eadf2748c98f2692df82ec4b51 (127.5.148.1:34125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ae79260e59a478ba024103d6d635cba: Network error: Client connection negotiation failed: client connection to 127.5.148.1:34125: connect: Connection refused (error 111)
I20260812 14:25:08.592446 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 21129
I20260812 14:25:08.601667 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 20673
I20260812 14:25:08.610610 5712 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 14:25:08.956058 5712 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 14:25:08.956204 5712 mini_ranger.cc:66] Stopping Ranger...
I20260812 14:25:09.302495 5712 mini_ranger.cc:68] Stopped Ranger
2026-08-12 14:25:09.302 UTC [19539] LOG: received smart shutdown request
2026-08-12 14:25:09.308 UTC [19539] LOG: background worker "logical replication launcher" (PID 19548) exited with exit code 1
2026-08-12 14:25:09.308 UTC [19543] LOG: shutting down
2026-08-12 14:25:09.474 UTC [19539] LOG: database system is shut down
2026-08-12T14:25:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50588 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:25:09 dist-test-slave-8w9v krb5kdc[21363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:25:09 dist-test-slave-8w9v krb5kdc[21363](info): set up 2 sockets
Aug 12 14:25:09 dist-test-slave-8w9v krb5kdc[21363](info): commencing operation
krb5kdc: starting...
W20260812 14:25:10.797179 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:25:10 dist-test-slave-8w9v krb5kdc[21363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:25:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:25:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:25:10.927321 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:43629
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:43629,127.5.148.61:35103,127.5.148.60:46859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:11.012730 21377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:11.012975 21377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:11.013012 21377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:11.014232 21377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:25:11.014303 21377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:11.014325 21377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:11.014348 21377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:25:11.014379 21377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:25:11.015918 21377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:43629,127.5.148.61:35103,127.5.148.60:46859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:43629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:11.016254 21377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:11.016507 21377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:11.021780 21388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:11.021785 21385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:11.022015 21383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:11.022042 21377 server_base.cc:1034] running on GCE node
I20260812 14:25:11.022475 21377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:11.022872 21377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:11.024025 21377 hybrid_clock.cc:648] HybridClock initialized: now 1786544711024011 us; error 31 us; skew 500 ppm
Aug 12 14:25:11 dist-test-slave-8w9v krb5kdc[21363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:11.026713 21377 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:25:11.027129 21377 webserver.cc:460] Webserver started at http://127.5.148.62:42771/ using document root <none> and password file <none>
I20260812 14:25:11.027343 21377 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:11.027383 21377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:11.027494 21377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:25:11.028621 21377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "595a19f5f94243beb541811491bb80e8"
format_stamp: "Formatted at 2026-08-12 14:25:11 on dist-test-slave-8w9v"
server_key: "3baa7ae5690b9a85fbcdffb52bb84549"
server_key_iv: "f59a39863019ec773a3a3eeb8b11e0d8"
server_key_version: "encryptionkey@0"
I20260812 14:25:11.028926 21377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "595a19f5f94243beb541811491bb80e8"
format_stamp: "Formatted at 2026-08-12 14:25:11 on dist-test-slave-8w9v"
server_key: "3baa7ae5690b9a85fbcdffb52bb84549"
server_key_iv: "f59a39863019ec773a3a3eeb8b11e0d8"
server_key_version: "encryptionkey@0"
I20260812 14:25:11.030210 21377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:25:11.030931 21396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:11.031127 21377 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:11.031217 21377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "595a19f5f94243beb541811491bb80e8"
format_stamp: "Formatted at 2026-08-12 14:25:11 on dist-test-slave-8w9v"
server_key: "3baa7ae5690b9a85fbcdffb52bb84549"
server_key_iv: "f59a39863019ec773a3a3eeb8b11e0d8"
server_key_version: "encryptionkey@0"
I20260812 14:25:11.031275 21377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:11.052837 21377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:11.055328 21377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:11.055451 21377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:11.059480 21377 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:43629
I20260812 14:25:11.059604 21454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:43629 every 8 connection(s)
I20260812 14:25:11.059957 21377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:25:11.060915 21455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:11.062485 21455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:25:11.063505 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 21377
I20260812 14:25:11.063654 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:25:11.063835 5712 external_mini_cluster.cc:1428] Setting key 118050cf4321b0afd1e7d59f01926f63
Aug 12 14:25:11 dist-test-slave-8w9v krb5kdc[21363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:11.073115 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.063151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:43629 (local address 127.0.0.1:48522)
0812 14:25:11.063304 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.063341 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:11.064019 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:11.064342 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:11.064345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:11.064641 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:11.064797 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.064801 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.065823 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.065824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.066513 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.066514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.067244 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.067245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.067317 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.067853 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:11.067861 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:11.069330 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:11.071543 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.071551 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.071553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.071748 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.072043 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.072044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.072044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.072186 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.072658 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:11.072664 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:11.072727 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 14:25:11.072803 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 14:25:11.072942 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260812 14:25:11.073190 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.063348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48522 (local address 127.5.148.62:43629)
0812 14:25:11.063492 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:11.063497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:11.063531 (+ 34us) server_negotiation.cc:407] Connection header received
0812 14:25:11.064220 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:11.064222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:11.064239 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:11.064334 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:11.064910 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.065706 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.066613 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.067131 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.067245 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.069427 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:11.069437 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:11.069438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:11.069446 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:11.071447 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.071840 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.071844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.071844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.071964 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.072263 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.072265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.072265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.072577 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:11.072677 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:11.072866 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 14:25:11.073033 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.5.148.61@KRBTEST.COM; defaulting to no policy
I20260812 14:25:11.074092 21455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:25:11.074474 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.074355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35103 (local address 127.0.0.1:34868)
0812 14:25:11.074404 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.074455 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260812 14:25:11.074615 21400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35103: connect: Connection refused (error 111)
I20260812 14:25:11.075067 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.074983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35103 (local address 127.0.0.1:34874)
0812 14:25:11.075027 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.075051 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 14:25:11.075187 21455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.61:35103: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35103: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.61.keytab.
Entry for principal kudu/127.5.148.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.61.keytab.
I20260812 14:25:11.094892 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.61:35103
--webserver_interface=127.5.148.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.61.keytab
--principal=kudu/127.5.148.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:43629,127.5.148.61:35103,127.5.148.60:46859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:25:11.124429 21455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } attempt: 1
I20260812 14:25:11.125005 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.124832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35103 (local address 127.0.0.1:34890)
0812 14:25:11.124911 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.124964 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 14:25:11.125411 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.125311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35103 (local address 127.0.0.1:34896)
0812 14:25:11.125358 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.125389 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 14:25:11.125530 21455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.61:35103: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35103: connect: Connection refused (error 111)
W20260812 14:25:11.181028 21463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:11.181241 21463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:11.181263 21463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:11.182431 21463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:25:11.182476 21463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:11.182492 21463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:11.182503 21463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:25:11.182556 21463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:25:11.184041 21463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:43629,127.5.148.61:35103,127.5.148.60:46859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.61.keytab
--principal=kudu/127.5.148.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.61:35103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.148.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:11.184338 21463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:11.184626 21463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:11.189934 21472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:11.190086 21463 server_base.cc:1034] running on GCE node
W20260812 14:25:11.189934 21470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:11.190018 21468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:11.190542 21463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:11.190834 21463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:11.192016 21463 hybrid_clock.cc:648] HybridClock initialized: now 1786544711192009 us; error 37 us; skew 500 ppm
I20260812 14:25:11.193704 21455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } attempt: 2
I20260812 14:25:11.194192 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.194024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35103 (local address 127.0.0.1:34902)
0812 14:25:11.194105 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.194159 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
Aug 12 14:25:11 dist-test-slave-8w9v krb5kdc[21363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:11.194615 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.194501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35103 (local address 127.0.0.1:34910)
0812 14:25:11.194554 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.194584 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260812 14:25:11.194736 21455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.61:35103: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35103: connect: Connection refused (error 111)
I20260812 14:25:11.194959 21463 init.cc:377] Logged in from keytab as kudu/127.5.148.61@KRBTEST.COM (short username kudu)
I20260812 14:25:11.195418 21463 webserver.cc:460] Webserver started at http://127.5.148.61:41371/ using document root <none> and password file <none>
I20260812 14:25:11.195655 21463 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:11.195724 21463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:11.195906 21463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:25:11.197069 21463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/data/instance:
uuid: "0f9534bfb3334a65905d0c11773faede"
format_stamp: "Formatted at 2026-08-12 14:25:11 on dist-test-slave-8w9v"
server_key: "7ce3c6b22fcf79beeeabe946664ba383"
server_key_iv: "f0aa717c0340c43bfebbae361c0ab4d1"
server_key_version: "encryptionkey@0"
I20260812 14:25:11.197453 21463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/wal/instance:
uuid: "0f9534bfb3334a65905d0c11773faede"
format_stamp: "Formatted at 2026-08-12 14:25:11 on dist-test-slave-8w9v"
server_key: "7ce3c6b22fcf79beeeabe946664ba383"
server_key_iv: "f0aa717c0340c43bfebbae361c0ab4d1"
server_key_version: "encryptionkey@0"
I20260812 14:25:11.198819 21463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:25:11.199574 21480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:11.199787 21463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:11.199895 21463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/wal
uuid: "0f9534bfb3334a65905d0c11773faede"
format_stamp: "Formatted at 2026-08-12 14:25:11 on dist-test-slave-8w9v"
server_key: "7ce3c6b22fcf79beeeabe946664ba383"
server_key_iv: "f0aa717c0340c43bfebbae361c0ab4d1"
server_key_version: "encryptionkey@0"
I20260812 14:25:11.199978 21463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:11.213766 21463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:11.215605 21463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:11.215778 21463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:11.219750 21463 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.61:35103
I20260812 14:25:11.219786 21540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.61:35103 every 8 connection(s)
I20260812 14:25:11.220220 21463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/data/info.pb
I20260812 14:25:11.220944 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 21463
I20260812 14:25:11.221124 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-1/wal/instance
I20260812 14:25:11.221282 21541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:11.221350 5712 external_mini_cluster.cc:1428] Setting key 56c9ec9805e55394c481c36c4c6189a9
I20260812 14:25:11.223325 21541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:25:11 dist-test-slave-8w9v krb5kdc[21363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.61@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.148.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.60@KRBTEST.COM" created.
I20260812 14:25:11.232826 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.224107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.5.148.62:43629)
0812 14:25:11.224211 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:11.224214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:11.224428 (+ 214us) server_negotiation.cc:407] Connection header received
0812 14:25:11.224938 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:11.224940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:11.224955 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:11.225020 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:11.225645 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.226143 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.227129 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.227388 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.227475 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.229895 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:11.229896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:11.229897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:11.229905 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:11.231343 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.231780 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.231782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.231782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.231851 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.232139 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.232140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.232141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.232268 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:11.232349 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:11.232642 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 14:25:11.232731 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 14:25:11.233026 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.224083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:43629 (local address 127.0.0.1:48532)
0812 14:25:11.224304 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.224339 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:11.224819 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:11.225095 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:11.225098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:11.225353 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:11.225540 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.225544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.226242 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.226243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.227027 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.227029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.227489 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.227490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.227616 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.228185 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:11.228193 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:11.229788 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:11.231451 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.231460 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.231463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.231699 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.231922 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.231923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.231924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.232047 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.232347 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:11.232357 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:11.232504 (+ 147us) client_negotiation.cc:769] Sending connection context
0812 14:25:11.232619 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 14:25:11.232785 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":127,"threads_started":1}
I20260812 14:25:11.233669 21541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:25:11 dist-test-slave-8w9v krb5kdc[21363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.61@KRBTEST.COM for kudu/127.5.148.61@KRBTEST.COM
I20260812 14:25:11.241745 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.233949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35103 (local address 127.0.0.1:34912)
0812 14:25:11.233989 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.234001 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:11.234054 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:11.234436 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:11.234436 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:11.234554 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:11.234648 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.234649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.235700 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.235701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.236288 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.236290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.236760 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.236761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.236832 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.237311 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:11.237317 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:11.238702 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:11.240597 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.240598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.240598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.240766 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.241001 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.241002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.241002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.241056 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.241367 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:11.241368 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:11.241408 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:25:11.241497 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 14:25:11.241605 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260812 14:25:11.241782 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.234070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34912 (local address 127.5.148.61:35103)
0812 14:25:11.234257 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:11.234263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:11.234288 (+ 25us) server_negotiation.cc:407] Connection header received
0812 14:25:11.234312 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:11.234314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:11.234329 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:11.234415 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:11.234735 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.235594 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.236369 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.236682 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.236764 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.238790 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:11.238801 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:11.238801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:11.238808 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:11.240480 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.240868 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.240871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.240872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.240932 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.241134 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.241135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.241135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.241283 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:11.241371 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:11.241533 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 14:25:11.241659 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260812 14:25:11.242530 21541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:25:11.242990 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.242791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:46859 (local address 127.0.0.1:40160)
0812 14:25:11.242906 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.242964 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260812 14:25:11.243136 21483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
I20260812 14:25:11.243618 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.243516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:46859 (local address 127.0.0.1:40176)
0812 14:25:11.243563 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.243588 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 14:25:11.243767 21541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.60:46859: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.60.keytab.
Entry for principal kudu/127.5.148.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.60.keytab.
I20260812 14:25:11.251775 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.60:46859
--webserver_interface=127.5.148.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.60.keytab
--principal=kudu/127.5.148.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:43629,127.5.148.61:35103,127.5.148.60:46859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:25:11.285892 21455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } attempt: 3
Aug 12 14:25:11 dist-test-slave-8w9v krb5kdc[21363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for kudu/127.5.148.61@KRBTEST.COM
I20260812 14:25:11.293023 21541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } attempt: 1
I20260812 14:25:11.293349 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.286407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.5.148.61:35103)
0812 14:25:11.286460 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:11.286462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:11.286500 (+ 38us) server_negotiation.cc:407] Connection header received
0812 14:25:11.286594 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:11.286595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:11.286607 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:11.286670 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:11.287027 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.287537 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.288293 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.288514 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.288602 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.290684 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:11.290685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:11.290686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:11.290700 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:11.292167 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.292530 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.292531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.292531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.292596 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.292800 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.292801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.292802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.292924 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:11.293001 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:11.293175 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 14:25:11.293259 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:25:11.293361 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.286331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35103 (local address 127.0.0.1:34928)
0812 14:25:11.286439 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.286454 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:11.286514 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:11.286690 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:11.286691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:11.286867 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:11.286933 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.286935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.287627 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.287628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.288204 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.288207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.288604 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.288606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.288697 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.289271 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:11.289276 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:11.290589 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:11.292264 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.292265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.292265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.292454 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.292671 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.292671 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.292672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.292736 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.293004 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:11.293006 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:11.293056 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 14:25:11.293135 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 14:25:11.293247 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 14:25:11.293490 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.293380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:46859 (local address 127.0.0.1:40186)
0812 14:25:11.293435 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.293462 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 14:25:11.293763 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.293700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:46859 (local address 127.0.0.1:40190)
0812 14:25:11.293728 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.293746 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260812 14:25:11.293771 21455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } has no permanent_uuid. Determining permanent_uuid...
W20260812 14:25:11.293834 21541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.60:46859: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
I20260812 14:25:11.294114 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.293995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:46859 (local address 127.0.0.1:40206)
0812 14:25:11.294057 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.294087 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 14:25:11.294492 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.294411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:46859 (local address 127.0.0.1:40220)
0812 14:25:11.294454 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.294477 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 14:25:11.294600 21455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.60:46859: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
I20260812 14:25:11.325752 21455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } attempt: 1
I20260812 14:25:11.326299 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.326157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:46859 (local address 127.0.0.1:40236)
0812 14:25:11.326229 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.326269 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 14:25:11.326725 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.326640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:46859 (local address 127.0.0.1:40252)
0812 14:25:11.326684 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.326709 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 14:25:11.326848 21455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.60:46859: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
W20260812 14:25:11.339511 21549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:11.339720 21549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:11.339743 21549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:11.340895 21549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:25:11.340935 21549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:11.340950 21549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:11.340962 21549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:25:11.340975 21549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:25:11.342394 21549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:43629,127.5.148.61:35103,127.5.148.60:46859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.60.keytab
--principal=kudu/127.5.148.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.60:46859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.148.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:11.342595 21549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:11.342768 21549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:11.347707 21560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:11.347707 21564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:11.347769 21558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:11.347877 21549 server_base.cc:1034] running on GCE node
I20260812 14:25:11.348241 21549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:11.348563 21549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:11.349746 21549 hybrid_clock.cc:648] HybridClock initialized: now 1786544711349728 us; error 36 us; skew 500 ppm
Aug 12 14:25:11 dist-test-slave-8w9v krb5kdc[21363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:11.352391 21549 init.cc:377] Logged in from keytab as kudu/127.5.148.60@KRBTEST.COM (short username kudu)
I20260812 14:25:11.352877 21549 webserver.cc:460] Webserver started at http://127.5.148.60:39479/ using document root <none> and password file <none>
I20260812 14:25:11.353113 21549 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:11.353159 21549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:11.353335 21549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:25:11.354420 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/data/instance:
uuid: "13e6c14ecc194d34b5a8b8c6afa8133a"
format_stamp: "Formatted at 2026-08-12 14:25:11 on dist-test-slave-8w9v"
server_key: "e95831855eb2829e6944dc7b5cb554a5"
server_key_iv: "df7c3246755eabefc1f79fc7844641d4"
server_key_version: "encryptionkey@0"
I20260812 14:25:11.354805 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/wal/instance:
uuid: "13e6c14ecc194d34b5a8b8c6afa8133a"
format_stamp: "Formatted at 2026-08-12 14:25:11 on dist-test-slave-8w9v"
server_key: "e95831855eb2829e6944dc7b5cb554a5"
server_key_iv: "df7c3246755eabefc1f79fc7844641d4"
server_key_version: "encryptionkey@0"
I20260812 14:25:11.356330 21549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:25:11.357210 21574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:11.357556 21549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:11.357684 21549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/wal
uuid: "13e6c14ecc194d34b5a8b8c6afa8133a"
format_stamp: "Formatted at 2026-08-12 14:25:11 on dist-test-slave-8w9v"
server_key: "e95831855eb2829e6944dc7b5cb554a5"
server_key_iv: "df7c3246755eabefc1f79fc7844641d4"
server_key_version: "encryptionkey@0"
I20260812 14:25:11.357762 21549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:11.361963 21541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } attempt: 2
I20260812 14:25:11.362380 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.362241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:46859 (local address 127.0.0.1:40260)
0812 14:25:11.362303 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.362344 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 14:25:11.362774 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.362697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:46859 (local address 127.0.0.1:40262)
0812 14:25:11.362735 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.362759 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 14:25:11.362874 21541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.60:46859: Network error: Client connection negotiation failed: client connection to 127.5.148.60:46859: connect: Connection refused (error 111)
I20260812 14:25:11.371855 21549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:11.374470 21549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:11.374635 21549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:11.379031 21549 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.60:46859
I20260812 14:25:11.379091 21636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.60:46859 every 8 connection(s)
I20260812 14:25:11.379458 21549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/data/info.pb
I20260812 14:25:11.380470 21637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:11.382133 21637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:25:11 dist-test-slave-8w9v krb5kdc[21363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.60@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:11.388186 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 21549
I20260812 14:25:11.388317 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786544547303129-5712-0/minicluster-data/master-2/wal/instance
I20260812 14:25:11.388568 5712 external_mini_cluster.cc:1428] Setting key c3721baf7498a8b4436ef651769f7e8f
I20260812 14:25:11.391445 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.382923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.5.148.62:43629)
0812 14:25:11.382997 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:11.382999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:11.383156 (+ 157us) server_negotiation.cc:407] Connection header received
0812 14:25:11.383641 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:11.383642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:11.383652 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:11.383714 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:11.384398 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.384884 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.385867 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.386069 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.386125 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.388499 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:11.388501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:11.388501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:11.388507 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:11.390030 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.390482 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.390483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.390484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.390547 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.390803 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.390804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.390804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.390925 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:11.390999 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:11.391266 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 14:25:11.391351 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:25:11.391655 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.382840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:43629 (local address 127.0.0.1:48544)
0812 14:25:11.383020 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.383066 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:11.383555 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:11.383768 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:11.383772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:11.384027 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:11.384315 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.384320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.384972 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.384973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.385766 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.385769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.386151 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.386153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.386270 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.386813 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:11.386822 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:11.388401 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:11.390135 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.390143 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.390145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.390397 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.390616 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.390617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.390618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.390728 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.390991 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:11.390999 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:11.391134 (+ 135us) client_negotiation.cc:769] Sending connection context
0812 14:25:11.391244 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 14:25:11.391413 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
I20260812 14:25:11.392333 21637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:25:11 dist-test-slave-8w9v krb5kdc[21363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.60@KRBTEST.COM
Aug 12 14:25:11 dist-test-slave-8w9v krb5kdc[21363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.60@KRBTEST.COM for kudu/127.5.148.61@KRBTEST.COM
I20260812 14:25:11.397827 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.389003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.5.148.60:46859)
0812 14:25:11.389400 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:11.389404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:11.389422 (+ 18us) server_negotiation.cc:407] Connection header received
0812 14:25:11.389466 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:11.389469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:11.389483 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:11.389565 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:11.390149 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.390828 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.391571 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.392026 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.392110 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.394722 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:11.394729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:11.394729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:11.394736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:11.396505 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.396907 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.396910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.396910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.396965 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.397168 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.397170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.397170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.397327 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:11.397487 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:11.397623 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 14:25:11.397699 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":101,"threads_started":1}
I20260812 14:25:11.399610 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.392807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34930 (local address 127.5.148.61:35103)
0812 14:25:11.392852 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:11.392854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:11.392863 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:25:11.392936 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:11.392937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:11.392945 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:11.392992 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:11.393324 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.393848 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.394598 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.394864 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.394924 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.396825 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:11.396827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:11.396827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:11.396833 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:11.398440 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.398822 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.398823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.398824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.398891 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.399108 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.399108 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.399108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.399210 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:11.399260 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:11.399445 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 14:25:11.399508 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:25:11.399617 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.392743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35103 (local address 127.0.0.1:34930)
0812 14:25:11.392802 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.392814 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:11.392869 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:11.393015 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:11.393017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:11.393178 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:11.393242 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.393244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.393968 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.393969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.394506 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.394508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.394940 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.394941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.395017 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.395304 (+ 287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:11.395309 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:11.396737 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:11.398534 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.398535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.398536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.398741 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.398968 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.398969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.398969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.399050 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.399279 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:11.399280 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:11.399331 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 14:25:11.399418 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 14:25:11.399502 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 14:25:11.400072 21637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:25:11.402031 21455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } attempt: 2
Aug 12 14:25:11 dist-test-slave-8w9v krb5kdc[21363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
Aug 12 14:25:11 dist-test-slave-8w9v krb5kdc[21363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.60@KRBTEST.COM for kudu/127.5.148.60@KRBTEST.COM
Aug 12 14:25:11 dist-test-slave-8w9v krb5kdc[21363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for kudu/127.5.148.60@KRBTEST.COM
I20260812 14:25:11.407428 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.398732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.5.148.62:43629)
0812 14:25:11.398796 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:11.398798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:11.398836 (+ 38us) server_negotiation.cc:407] Connection header received
0812 14:25:11.398910 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:11.398911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:11.398922 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:11.398972 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:11.399589 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.400267 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.401140 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.401423 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.401490 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.404222 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:11.404224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:11.404224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:11.404231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:11.405860 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.406266 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.406268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.406268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.406323 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.406596 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.406597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.406598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.406714 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:11.406946 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:11.407279 (+ 333us) server_negotiation.cc:299] Negotiation successful
0812 14:25:11.407359 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:25:11.409817 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.400374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:46859 (local address 127.0.0.1:40278)
0812 14:25:11.400416 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.400425 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:11.400459 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:11.400561 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:11.400562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:11.400709 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:11.400742 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.400744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.401496 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.401497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.402245 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.402248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.402954 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.402955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.403061 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.403474 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:11.403479 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:11.404907 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:11.408019 (+ 3112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.408019 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.408020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.408246 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.408434 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.408435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.408436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.408510 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.408808 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:11.408809 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:11.408846 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 14:25:11.408906 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 14:25:11.409704 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 14:25:11.409955 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.400413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.5.148.60:46859)
0812 14:25:11.400447 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:11.400448 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:11.400456 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:25:11.400507 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:11.400508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:11.400515 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:11.400558 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:11.400809 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.401403 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.402539 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.402858 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.402938 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.405043 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:11.405045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:11.405046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:11.405053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:11.407949 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.408316 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.408317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.408317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.408375 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.408589 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.408590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.408591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.408723 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:11.408785 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:11.408996 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 14:25:11.409804 (+ 808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":719,"server-negotiator.queue_time_us":17}
I20260812 14:25:11.410770 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.402313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:46859 (local address 127.0.0.1:40280)
0812 14:25:11.402363 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.402372 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:11.402414 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:11.402677 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:11.402678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:11.402912 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:11.402955 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.402957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.403875 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.403876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.404692 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.404695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.405157 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.405159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.405227 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.405644 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:11.405649 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:11.407500 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:11.409637 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.409638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.409639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.409853 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.410095 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.410096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.410096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.410157 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.410453 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:11.410454 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:11.410501 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 14:25:11.410581 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 14:25:11.410673 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 14:25:11.410852 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.402350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.5.148.60:46859)
0812 14:25:11.402557 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:11.402559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:11.402565 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:25:11.402581 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:11.402581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:11.402589 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:11.402672 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:11.403019 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.403780 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.404784 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.405102 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.405151 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.407612 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:11.407614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:11.407614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:11.407622 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:11.409556 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.409945 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.409947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.409947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.410041 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.410224 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.410225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.410225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.410369 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:11.410440 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:11.410630 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 14:25:11.410708 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":135,"threads_started":1}
I20260812 14:25:11.411863 21637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a: Bootstrap starting.
I20260812 14:25:11.412333 21455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8: Bootstrap starting.
I20260812 14:25:11.412716 21637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:11.413077 21455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:11.413084 21637 log.cc:824] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a: Log is configured to *not* fsync() on all Append() calls
Aug 12 14:25:11 dist-test-slave-8w9v krb5kdc[21363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.61@KRBTEST.COM
I20260812 14:25:11.413429 21455 log.cc:824] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:11.414052 21637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a: No bootstrap required, opened a new log
I20260812 14:25:11.414456 21455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8: No bootstrap required, opened a new log
I20260812 14:25:11.416287 21455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } }
I20260812 14:25:11.416452 21455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:11.416491 21455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 595a19f5f94243beb541811491bb80e8, State: Initialized, Role: FOLLOWER
I20260812 14:25:11.416397 21637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } }
I20260812 14:25:11.416558 21637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:11.416603 21637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e6c14ecc194d34b5a8b8c6afa8133a, State: Initialized, Role: FOLLOWER
I20260812 14:25:11.416733 21637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } }
I20260812 14:25:11.416740 21455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } }
I20260812 14:25:11.417097 21455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:25:11.417097 21637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:25:11.417317 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.408542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.5.148.61:35103)
0812 14:25:11.408588 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:11.408590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:11.408621 (+ 31us) server_negotiation.cc:407] Connection header received
0812 14:25:11.408646 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:11.408647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:11.408658 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:11.408710 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:11.409356 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.409889 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.411005 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.411309 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.411354 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.414005 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:11.414006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:11.414007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:11.414015 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:11.415774 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.416319 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.416320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.416321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.416393 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.416629 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.416630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.416630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.416786 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:11.417025 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:11.417126 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 14:25:11.417217 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260812 14:25:11.420936 21673 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260812 14:25:11.420981 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260812 14:25:11.421058 21673 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:25:11.419626 21549 master_runner.cc:186] Error getting master registration for 127.5.148.62:43629: OK, instance_id { permanent_uuid: "595a19f5f94243beb541811491bb80e8" instance_seqno: 1786544711058307 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:25:11.422940 21549 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:25:11.421360 21377 master_runner.cc:186] Error getting master registration for 127.5.148.61:35103: OK, instance_id { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" instance_seqno: 1786544711218697 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:25:11.423848 21377 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:25:11.424271 21647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } } }
I20260812 14:25:11.424494 21647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:25:11.425397 21663 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:25:11.425500 21663 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:25:11.425660 21649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } } }
I20260812 14:25:11.425789 21649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:25:11.431783 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.422630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.5.148.62:43629)
0812 14:25:11.422684 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:11.422685 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:11.423330 (+ 645us) server_negotiation.cc:407] Connection header received
0812 14:25:11.423456 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:11.423457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:11.423469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:11.423504 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:11.424449 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.425198 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.426248 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.426540 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.426583 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.428602 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:11.428603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:11.428604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:11.428610 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:11.430401 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.430865 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.430867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.430867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.430938 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.431196 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.431197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.431198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.431322 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:11.431446 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:11.431576 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 14:25:11.431666 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:25:11.433383 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.423064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.5.148.61:35103)
0812 14:25:11.423102 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:11.423104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:11.423724 (+ 620us) server_negotiation.cc:407] Connection header received
0812 14:25:11.423822 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:11.423823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:11.423835 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:11.423865 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:11.425519 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.426219 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.427106 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.427375 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.427454 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.429372 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:11.429374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:11.429374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:11.429383 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:11.431769 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.432482 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.432484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.432484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.432544 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.432750 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.432751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.432751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.432880 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:11.432983 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:11.433187 (+ 204us) server_negotiation.cc:299] Negotiation successful
0812 14:25:11.433276 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:25:11.436676 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.423426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40290 (local address 127.5.148.60:46859)
0812 14:25:11.423464 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:11.423466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:11.424673 (+ 1207us) server_negotiation.cc:407] Connection header received
0812 14:25:11.424760 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:11.424761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:11.424772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:11.424809 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:11.427148 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.427845 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.428920 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.429272 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.429343 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.430764 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:11.430765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:11.430766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:11.430772 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:11.432835 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.433211 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.433213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.433213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.433278 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.433494 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.433495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.433495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.436222 (+ 2727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:11.436369 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:11.436481 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 14:25:11.436580 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:25:11.454031 21541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } attempt: 3
Aug 12 14:25:11 dist-test-slave-8w9v krb5kdc[21363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.61@KRBTEST.COM for kudu/127.5.148.60@KRBTEST.COM
I20260812 14:25:11.463191 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.454364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:46859 (local address 127.0.0.1:40300)
0812 14:25:11.454421 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:11.454434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:11.454473 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:11.454657 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:11.454658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:11.454884 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:11.454942 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.454944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.455931 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.455932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.456785 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:11.456789 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.457309 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.457310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:11.457373 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.457952 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:11.457958 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:11.459644 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:11.461946 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.461948 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.461949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.462207 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.462447 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:11.462448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:11.462449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:11.462526 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:11.462831 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:11.462832 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:11.462886 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 14:25:11.462968 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 14:25:11.463078 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 14:25:11.463200 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:11.454440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.5.148.60:46859)
0812 14:25:11.454504 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:11.454506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:11.454517 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:25:11.454546 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:11.454546 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:11.454558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:11.454635 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:11.455042 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.455825 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.456888 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:11.457210 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:11.457287 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:11.459750 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:11.459763 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:11.459763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:11.459772 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:11.461848 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.462299 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.462300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.462301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.462366 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:11.462603 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:11.462604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:11.462604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:11.462748 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:11.462828 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:11.463006 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 14:25:11.463103 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:25:11.465564 21541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede: Bootstrap starting.
I20260812 14:25:11.466408 21541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:11.466732 21541 log.cc:824] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:11.467751 21541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede: No bootstrap required, opened a new log
I20260812 14:25:11.470073 21541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } }
I20260812 14:25:11.470839 21541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:11.470973 21541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9534bfb3334a65905d0c11773faede, State: Initialized, Role: FOLLOWER
I20260812 14:25:11.471179 21541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } }
I20260812 14:25:11.471627 21541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:25:11.471591 21684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } } }
I20260812 14:25:11.471860 21684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:25:11.473721 21695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:25:11.473856 21695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:25:11.476178 21463 master_runner.cc:263] No leader master found from master 0f9534bfb3334a65905d0c11773faede
I20260812 14:25:11.476238 21463 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:25:11.549166 21647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:25:11.549330 21647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } }
I20260812 14:25:11.549602 21647 leader_election.cc:288] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 595a19f5f94243beb541811491bb80e8 (127.5.148.62:43629), 0f9534bfb3334a65905d0c11773faede (127.5.148.61:35103)
I20260812 14:25:11.550380 21509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9534bfb3334a65905d0c11773faede" is_pre_election: true
I20260812 14:25:11.550580 21509 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e6c14ecc194d34b5a8b8c6afa8133a in term 0.
I20260812 14:25:11.550918 21427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595a19f5f94243beb541811491bb80e8" is_pre_election: true
I20260812 14:25:11.551081 21427 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e6c14ecc194d34b5a8b8c6afa8133a in term 0.
I20260812 14:25:11.551391 21580 leader_election.cc:302] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e6c14ecc194d34b5a8b8c6afa8133a, 595a19f5f94243beb541811491bb80e8; no voters:
I20260812 14:25:11.551757 21647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:25:11.551822 21647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:25:11.551849 21647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:11.553340 21647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } }
I20260812 14:25:11.553479 21647 leader_election.cc:288] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [CANDIDATE]: Term 1 election: Requested vote from peers 595a19f5f94243beb541811491bb80e8 (127.5.148.62:43629), 0f9534bfb3334a65905d0c11773faede (127.5.148.61:35103)
I20260812 14:25:11.553860 21509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9534bfb3334a65905d0c11773faede"
I20260812 14:25:11.553951 21509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:11.554889 21509 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e6c14ecc194d34b5a8b8c6afa8133a in term 1.
I20260812 14:25:11.555145 21580 leader_election.cc:302] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9534bfb3334a65905d0c11773faede, 13e6c14ecc194d34b5a8b8c6afa8133a; no voters:
I20260812 14:25:11.555305 21427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595a19f5f94243beb541811491bb80e8"
I20260812 14:25:11.555402 21427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8 [term 0 FOLLOWER]: Advancing to term 1
W20260812 14:25:11.556245 21404 tablet.cc:2369] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:25:11.556403 21427 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e6c14ecc194d34b5a8b8c6afa8133a in term 1.
I20260812 14:25:11.556730 21647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:25:11.556932 21647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [term 1 LEADER]: Becoming Leader. State: Replica: 13e6c14ecc194d34b5a8b8c6afa8133a, State: Running, Role: LEADER
I20260812 14:25:11.557026 21647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } }
I20260812 14:25:11.557452 21647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13e6c14ecc194d34b5a8b8c6afa8133a. Latest consensus state: current_term: 1 leader_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } } }
I20260812 14:25:11.557533 21647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:11.559096 21700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:25:11.559373 21700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:25:11.561821 21509 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede [term 1 FOLLOWER]: Refusing update from remote peer 13e6c14ecc194d34b5a8b8c6afa8133a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:25:11.562234 21647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:25:11.563614 21684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13e6c14ecc194d34b5a8b8c6afa8133a. Latest consensus state: current_term: 1 leader_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } } }
I20260812 14:25:11.563714 21684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:25:11.564280 21427 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8 [term 1 FOLLOWER]: Refusing update from remote peer 13e6c14ecc194d34b5a8b8c6afa8133a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:25:11.564628 21647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [LEADER]: Connected to new peer: Peer: permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:25:11.565959 21649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13e6c14ecc194d34b5a8b8c6afa8133a. Latest consensus state: current_term: 1 leader_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } } }
I20260812 14:25:11.566061 21649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:25:11.567528 21698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } } }
I20260812 14:25:11.567631 21698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:11.568269 21649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } } }
I20260812 14:25:11.568361 21649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:25:11.568661 21702 mvcc.cc:205] Tried to move back new op lower bound from 7317687138554609664 to 7317687138538237952. Current Snapshot: MvccSnapshot[applied={T|T < 7317687138554609664}]
I20260812 14:25:11.568917 21705 mvcc.cc:205] Tried to move back new op lower bound from 7317687138554609664 to 7317687138538237952. Current Snapshot: MvccSnapshot[applied={T|T < 7317687138554609664}]
I20260812 14:25:11.569847 21647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } } }
I20260812 14:25:11.569936 21647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:11.570360 21684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a19f5f94243beb541811491bb80e8" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 43629 } } peers { permanent_uuid: "0f9534bfb3334a65905d0c11773faede" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35103 } } peers { permanent_uuid: "13e6c14ecc194d34b5a8b8c6afa8133a" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 46859 } } }
I20260812 14:25:11.570458 21684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:25:11.571219 21703 mvcc.cc:205] Tried to move back new op lower bound from 7317687138554609664 to 7317687138538237952. Current Snapshot: MvccSnapshot[applied={T|T < 7317687138554609664}]
I20260812 14:25:11.572297 21700 catalog_manager.cc:1347] Generated new cluster ID: f60b892b95ec4aaf9838e5f7ed851eb1
I20260812 14:25:11.572352 21700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:25:11.598251 21700 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:25:11.599133 21700 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:25:11.610505 21700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13e6c14ecc194d34b5a8b8c6afa8133a: Generated new TSK 0
I20260812 14:25:11.610762 21700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:25:12.421695 21673 catalog_manager.cc:1259] Loaded cluster ID: f60b892b95ec4aaf9838e5f7ed851eb1
I20260812 14:25:12.421787 21673 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8: loading cluster ID for follower catalog manager: success
I20260812 14:25:12.423205 21673 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8: acquiring CA information for follower catalog manager: success
I20260812 14:25:12.423604 21673 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 595a19f5f94243beb541811491bb80e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:25:12.474382 21695 catalog_manager.cc:1259] Loaded cluster ID: f60b892b95ec4aaf9838e5f7ed851eb1
I20260812 14:25:12.474471 21695 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede: loading cluster ID for follower catalog manager: success
I20260812 14:25:12.475328 21695 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede: acquiring CA information for follower catalog manager: success
I20260812 14:25:12.475667 21695 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0f9534bfb3334a65905d0c11773faede: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:25:12.677253 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 21377
I20260812 14:25:12.686144 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 21463
I20260812 14:25:12.694761 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 21549
2026-08-12T14:25:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3096 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:25:12 dist-test-slave-8w9v krb5kdc[21716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:25:12 dist-test-slave-8w9v krb5kdc[21716](info): set up 2 sockets
Aug 12 14:25:12 dist-test-slave-8w9v krb5kdc[21716](info): commencing operation
krb5kdc: starting...
W20260812 14:25:13.859345 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:25:13 dist-test-slave-8w9v krb5kdc[21716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:25:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:25:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:25:13.993114 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:39399
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:39399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:14.080535 21732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:14.080842 21732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:14.080889 21732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:14.082214 21732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:25:14.082304 21732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:14.082335 21732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:14.082371 21732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:25:14.082397 21732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:25:14.084348 21732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:39399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:39399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:14.084685 21732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:14.084976 21732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:14.090400 21738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:14.090451 21739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:14.090454 21741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:14.090623 21732 server_base.cc:1034] running on GCE node
I20260812 14:25:14.090885 21732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:14.091199 21732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:14.092344 21732 hybrid_clock.cc:648] HybridClock initialized: now 1786544714092328 us; error 30 us; skew 500 ppm
Aug 12 14:25:14 dist-test-slave-8w9v krb5kdc[21716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544714, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:14.095120 21732 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:25:14.095628 21732 webserver.cc:460] Webserver started at http://127.5.148.62:36209/ using document root <none> and password file <none>
I20260812 14:25:14.095893 21732 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:14.095973 21732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:14.096194 21732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:25:14.097301 21732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "348fd10b578a45a38ff02b80fe2f0b6b"
format_stamp: "Formatted at 2026-08-12 14:25:14 on dist-test-slave-8w9v"
server_key: "ed423b05e57c3007ffdb351de2770dda"
server_key_iv: "3dad536a9254018131b34d10006b0324"
server_key_version: "encryptionkey@0"
I20260812 14:25:14.097678 21732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "348fd10b578a45a38ff02b80fe2f0b6b"
format_stamp: "Formatted at 2026-08-12 14:25:14 on dist-test-slave-8w9v"
server_key: "ed423b05e57c3007ffdb351de2770dda"
server_key_iv: "3dad536a9254018131b34d10006b0324"
server_key_version: "encryptionkey@0"
I20260812 14:25:14.099138 21732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:25:14.099916 21748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:14.100091 21732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:25:14.100229 21732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "348fd10b578a45a38ff02b80fe2f0b6b"
format_stamp: "Formatted at 2026-08-12 14:25:14 on dist-test-slave-8w9v"
server_key: "ed423b05e57c3007ffdb351de2770dda"
server_key_iv: "3dad536a9254018131b34d10006b0324"
server_key_version: "encryptionkey@0"
I20260812 14:25:14.100294 21732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:14.107950 21732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:14.110289 21732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:14.110414 21732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:14.114830 21732 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:39399
I20260812 14:25:14.114867 21813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:39399 every 8 connection(s)
I20260812 14:25:14.115265 21732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:25:14.116315 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:14.118727 21814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b: Bootstrap starting.
I20260812 14:25:14.119313 21814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:14.119391 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 21732
I20260812 14:25:14.119522 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:25:14.119725 5712 external_mini_cluster.cc:1428] Setting key c768112fcf561a2dd5f11f37c85d27f0
I20260812 14:25:14.119834 21814 log.cc:824] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:14.120730 21814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b: No bootstrap required, opened a new log
I20260812 14:25:14.122983 21814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348fd10b578a45a38ff02b80fe2f0b6b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39399 } }
I20260812 14:25:14.123165 21814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:14.123221 21814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348fd10b578a45a38ff02b80fe2f0b6b, State: Initialized, Role: FOLLOWER
I20260812 14:25:14.123395 21814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348fd10b578a45a38ff02b80fe2f0b6b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39399 } }
I20260812 14:25:14.123507 21814 raft_consensus.cc:388] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:25:14.123557 21814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:25:14.123611 21814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:14.124286 21814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348fd10b578a45a38ff02b80fe2f0b6b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39399 } }
I20260812 14:25:14.124418 21814 leader_election.cc:302] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 348fd10b578a45a38ff02b80fe2f0b6b; no voters:
I20260812 14:25:14.124626 21814 leader_election.cc:288] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:25:14.124658 21819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:25:14.124851 21819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [term 1 LEADER]: Becoming Leader. State: Replica: 348fd10b578a45a38ff02b80fe2f0b6b, State: Running, Role: LEADER
I20260812 14:25:14.124951 21819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348fd10b578a45a38ff02b80fe2f0b6b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39399 } }
I20260812 14:25:14.125015 21814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:25:14.125308 21819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "348fd10b578a45a38ff02b80fe2f0b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348fd10b578a45a38ff02b80fe2f0b6b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39399 } } }
I20260812 14:25:14.125339 21821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 348fd10b578a45a38ff02b80fe2f0b6b. Latest consensus state: current_term: 1 leader_uuid: "348fd10b578a45a38ff02b80fe2f0b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348fd10b578a45a38ff02b80fe2f0b6b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39399 } } }
I20260812 14:25:14.125468 21819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:14.125715 21821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:14.126559 21825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 12 14:25:14 dist-test-slave-8w9v krb5kdc[21716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:14.127511 21825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:25:14.129524 21825 catalog_manager.cc:1347] Generated new cluster ID: c52e5e7658404fab9c810ded5b21f633
I20260812 14:25:14.129581 21825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:25:14.131889 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:14.120255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.5.148.62:39399)
0812 14:25:14.120595 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:14.120603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:14.120628 (+ 25us) server_negotiation.cc:407] Connection header received
0812 14:25:14.121137 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:14.121144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:14.121196 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:14.121350 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:14.122029 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:14.122956 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:14.123934 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:14.124481 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:14.124610 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:14.127670 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:14.127674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:14.127674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:14.127688 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:14.130014 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:14.130515 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:14.130519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:14.130519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:14.130636 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:14.130878 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:14.130879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:14.130879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:14.131195 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:14.131362 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:14.131510 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 14:25:14.131654 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":92,"threads_started":1}
I20260812 14:25:14.132963 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:25:14.135794 21825 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:25:14.136596 21825 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:25:14.143836 21825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 348fd10b578a45a38ff02b80fe2f0b6b: Generated new TSK 0
I20260812 14:25:14.144050 21825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:25:14.187165 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 21732
2026-08-12T14:25:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1491 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:25:14 dist-test-slave-8w9v krb5kdc[21844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:25:14 dist-test-slave-8w9v krb5kdc[21844](info): set up 2 sockets
Aug 12 14:25:14 dist-test-slave-8w9v krb5kdc[21844](info): commencing operation
krb5kdc: starting...
W20260812 14:25:15.315333 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:25:15 dist-test-slave-8w9v krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:25:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:25:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:25:15.446897 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:37885
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:37885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:15.532351 21859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:15.532615 21859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:15.532670 21859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:15.533855 21859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:25:15.533929 21859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:15.533953 21859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:15.533974 21859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:25:15.534004 21859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:25:15.535509 21859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:37885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:37885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:15.535799 21859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:15.536059 21859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:15.541172 21866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:15.541422 21859 server_base.cc:1034] running on GCE node
W20260812 14:25:15.541183 21865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:15.541188 21868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:15.541828 21859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:15.542141 21859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:15.543293 21859 hybrid_clock.cc:648] HybridClock initialized: now 1786544715543276 us; error 29 us; skew 500 ppm
Aug 12 14:25:15 dist-test-slave-8w9v krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544715, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:15.551440 21859 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:25:15.551959 21859 webserver.cc:460] Webserver started at http://127.5.148.62:35021/ using document root <none> and password file <none>
I20260812 14:25:15.552219 21859 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:15.552271 21859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:15.552374 21859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:25:15.553444 21859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "ca6e04394f36473c94903b255ac7a270"
format_stamp: "Formatted at 2026-08-12 14:25:15 on dist-test-slave-8w9v"
server_key: "02b79de3c5a07bd30631595f0a04cb91"
server_key_iv: "874d1720ebd2192587998063808bbbcd"
server_key_version: "encryptionkey@0"
I20260812 14:25:15.553753 21859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "ca6e04394f36473c94903b255ac7a270"
format_stamp: "Formatted at 2026-08-12 14:25:15 on dist-test-slave-8w9v"
server_key: "02b79de3c5a07bd30631595f0a04cb91"
server_key_iv: "874d1720ebd2192587998063808bbbcd"
server_key_version: "encryptionkey@0"
I20260812 14:25:15.555150 21859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:25:15.555917 21878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:15.556193 21859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:15.556296 21859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "ca6e04394f36473c94903b255ac7a270"
format_stamp: "Formatted at 2026-08-12 14:25:15 on dist-test-slave-8w9v"
server_key: "02b79de3c5a07bd30631595f0a04cb91"
server_key_iv: "874d1720ebd2192587998063808bbbcd"
server_key_version: "encryptionkey@0"
I20260812 14:25:15.556384 21859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:15.568465 21859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:15.570536 21859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:15.570798 21859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:15.575487 21859 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:37885
I20260812 14:25:15.575551 21938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:37885 every 8 connection(s)
I20260812 14:25:15.575894 21859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:25:15.576977 21939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:15.579324 21939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: Bootstrap starting.
I20260812 14:25:15.579953 21939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:15.580320 21939 log.cc:824] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:15.581068 21939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: No bootstrap required, opened a new log
I20260812 14:25:15.582707 21939 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } }
I20260812 14:25:15.582862 21939 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:15.582913 21939 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6e04394f36473c94903b255ac7a270, State: Initialized, Role: FOLLOWER
I20260812 14:25:15.583088 21939 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } }
I20260812 14:25:15.583191 21939 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:25:15.583241 21939 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:25:15.583300 21939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:15.583384 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 21859
I20260812 14:25:15.583513 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:25:15.583699 5712 external_mini_cluster.cc:1428] Setting key 289db7c9ef8a51f92c1b7375202ee1bb
I20260812 14:25:15.584578 21939 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } }
I20260812 14:25:15.584707 21939 leader_election.cc:302] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca6e04394f36473c94903b255ac7a270; no voters:
I20260812 14:25:15.584916 21939 leader_election.cc:288] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:25:15.584937 21944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:25:15.585057 21944 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 1 LEADER]: Becoming Leader. State: Replica: ca6e04394f36473c94903b255ac7a270, State: Running, Role: LEADER
I20260812 14:25:15.585198 21939 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:25:15.585210 21944 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } }
I20260812 14:25:15.585698 21946 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca6e04394f36473c94903b255ac7a270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } } }
I20260812 14:25:15.585799 21946 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:15.585716 21945 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca6e04394f36473c94903b255ac7a270. Latest consensus state: current_term: 1 leader_uuid: "ca6e04394f36473c94903b255ac7a270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } } }
I20260812 14:25:15.586004 21945 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:15.586745 21952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:25:15.587518 21952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:25:15.589257 21952 catalog_manager.cc:1347] Generated new cluster ID: 949fa5c383d24fc9b3ed9ba771483732
I20260812 14:25:15.589313 21952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:25:15 dist-test-slave-8w9v krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:15.594909 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:15.584262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55844 (local address 127.5.148.62:37885)
0812 14:25:15.584498 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:15.584509 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:15.584546 (+ 37us) server_negotiation.cc:407] Connection header received
0812 14:25:15.584644 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:15.584650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:15.584714 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:15.584893 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:15.585794 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:15.587271 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:15.588170 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:15.588805 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:15.588937 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:15.591420 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:15.591424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:15.591424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:15.591436 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:15.593250 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:15.593676 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:15.593678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:15.593679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:15.593787 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:15.594012 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:15.594013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:15.594013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:15.594256 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:15.594446 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:15.594515 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 14:25:15.594718 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":148,"threads_started":1}
I20260812 14:25:15.608987 21952 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:25:15.609611 21952 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:25:15.622174 21952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: Generated new TSK 0
I20260812 14:25:15.622435 21952 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:25:15.659322 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:37885
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:15.759820 21970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:15.760031 21970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:15.760056 21970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:15.761276 21970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:15.761324 21970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:15.761386 21970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:25:15.762899 21970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:15.763157 21970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:15.763407 21970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:15.768682 21976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:15.768687 21977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:15.768692 21980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:15.768858 21970 server_base.cc:1034] running on GCE node
I20260812 14:25:15.769104 21970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:15.769385 21970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:15.770558 21970 hybrid_clock.cc:648] HybridClock initialized: now 1786544715770552 us; error 36 us; skew 500 ppm
Aug 12 14:25:15 dist-test-slave-8w9v krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544715, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:15.773290 21970 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:25:15.773746 21970 webserver.cc:460] Webserver started at http://127.5.148.1:40083/ using document root <none> and password file <none>
I20260812 14:25:15.773973 21970 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:15.774019 21970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:15.774107 21970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:25:15.775224 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "2d5966afdbdc46e8b5738cf0218cbcc4"
format_stamp: "Formatted at 2026-08-12 14:25:15 on dist-test-slave-8w9v"
server_key: "309cb3e478583a4aaa2a4e190a1a88c6"
server_key_iv: "68d1ba2c874db918302e61656fc81a01"
server_key_version: "encryptionkey@0"
I20260812 14:25:15.775600 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "2d5966afdbdc46e8b5738cf0218cbcc4"
format_stamp: "Formatted at 2026-08-12 14:25:15 on dist-test-slave-8w9v"
server_key: "309cb3e478583a4aaa2a4e190a1a88c6"
server_key_iv: "68d1ba2c874db918302e61656fc81a01"
server_key_version: "encryptionkey@0"
I20260812 14:25:15.777065 21970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:25:15.777773 21987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:15.777983 21970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:15.778071 21970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "2d5966afdbdc46e8b5738cf0218cbcc4"
format_stamp: "Formatted at 2026-08-12 14:25:15 on dist-test-slave-8w9v"
server_key: "309cb3e478583a4aaa2a4e190a1a88c6"
server_key_iv: "68d1ba2c874db918302e61656fc81a01"
server_key_version: "encryptionkey@0"
I20260812 14:25:15.778126 21970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:15.796725 21970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:15.798921 21970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:15.799045 21970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:15.799253 21970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:15.799564 21970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:15.799597 21970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:15.799620 21970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:15.799633 21970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:15.805653 21970 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:46651
I20260812 14:25:15.805739 22116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:46651 every 8 connection(s)
I20260812 14:25:15.806078 21970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:25:15.806399 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 21970
I20260812 14:25:15.806512 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:25:15.806723 5712 external_mini_cluster.cc:1428] Setting key 1ab699ce52721060800064332030a2ec
Aug 12 14:25:15 dist-test-slave-8w9v krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544715, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
I20260812 14:25:15.817006 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:15.806941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:56685 (local address 127.5.148.62:37885)
0812 14:25:15.807019 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:15.807022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:15.807264 (+ 242us) server_negotiation.cc:407] Connection header received
0812 14:25:15.807989 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:15.807990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:15.808009 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:15.808070 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:15.809479 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:15.810000 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:15.811429 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:15.811644 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:15.811723 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:15.814008 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:15.814010 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:15.814010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:15.814018 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:15.815492 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:15.815925 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:15.815927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:15.815927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:15.816000 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:15.816331 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:15.816333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:15.816333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:15.816501 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:15.816589 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:15.816846 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 14:25:15.816939 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 14:25:15.817135 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:15.806930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:56685)
0812 14:25:15.807136 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:15.807175 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:15.807883 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:15.808169 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:15.808174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:15.809046 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:15.809346 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:15.809353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:15.810102 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:15.810103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:15.811326 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:15.811330 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:15.811732 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:15.811734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:15.811877 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:15.812489 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:15.812498 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:15.813899 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:15.815599 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:15.815607 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:15.815610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:15.815838 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:15.816072 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:15.816073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:15.816073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:15.816234 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:15.816582 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:15.816588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:15.816711 (+ 123us) client_negotiation.cc:769] Sending connection context
0812 14:25:15.816797 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 14:25:15.816933 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20260812 14:25:15.817744 22117 heartbeater.cc:347] Connected to a master server at 127.5.148.62:37885
I20260812 14:25:15.817868 22117 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:15.818037 22117 heartbeater.cc:511] Master 127.5.148.62:37885 requested a full tablet report, sending...
I20260812 14:25:15.818682 21899 ts_manager.cc:194] Registered new tserver with Master: 2d5966afdbdc46e8b5738cf0218cbcc4 (127.5.148.1:46651)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
I20260812 14:25:15.819608 21899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:56685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:25:15.835378 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:37885
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:15.919569 22127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:15.919776 22127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:15.919800 22127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:15.921052 22127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:15.921101 22127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:15.921128 22127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:25:15.922636 22127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:15.923045 22127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:15.923362 22127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:15.929088 22133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:15.929102 22135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:15.929160 22127 server_base.cc:1034] running on GCE node
W20260812 14:25:15.929105 22132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:15.929677 22127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:15.930011 22127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:15.931193 22127 hybrid_clock.cc:648] HybridClock initialized: now 1786544715931179 us; error 48 us; skew 500 ppm
Aug 12 14:25:15 dist-test-slave-8w9v krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544715, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:15.934027 22127 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:25:15.934545 22127 webserver.cc:460] Webserver started at http://127.5.148.2:43999/ using document root <none> and password file <none>
I20260812 14:25:15.934821 22127 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:15.934895 22127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:15.935050 22127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:25:15.936285 22127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "b8252fe49ffc4e34b1c65dde56faf03d"
format_stamp: "Formatted at 2026-08-12 14:25:15 on dist-test-slave-8w9v"
server_key: "d4714e8e7602048df87c8f8d8362e91c"
server_key_iv: "2e79d138829b0a5523c4b8769e540c7c"
server_key_version: "encryptionkey@0"
I20260812 14:25:15.936683 22127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "b8252fe49ffc4e34b1c65dde56faf03d"
format_stamp: "Formatted at 2026-08-12 14:25:15 on dist-test-slave-8w9v"
server_key: "d4714e8e7602048df87c8f8d8362e91c"
server_key_iv: "2e79d138829b0a5523c4b8769e540c7c"
server_key_version: "encryptionkey@0"
I20260812 14:25:15.938258 22127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:25:15.939077 22143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:15.939258 22127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:15.939366 22127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "b8252fe49ffc4e34b1c65dde56faf03d"
format_stamp: "Formatted at 2026-08-12 14:25:15 on dist-test-slave-8w9v"
server_key: "d4714e8e7602048df87c8f8d8362e91c"
server_key_iv: "2e79d138829b0a5523c4b8769e540c7c"
server_key_version: "encryptionkey@0"
I20260812 14:25:15.939429 22127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:15.960790 22127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:15.963169 22127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:15.963346 22127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:15.963603 22127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:15.963945 22127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:15.964003 22127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:15.964051 22127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:15.964092 22127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:15.970280 22127 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:39335
I20260812 14:25:15.970332 22275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:39335 every 8 connection(s)
I20260812 14:25:15.970654 22127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:25:15.971747 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 22127
I20260812 14:25:15.971871 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:25:15.972066 5712 external_mini_cluster.cc:1428] Setting key fe5b64a45c282ea7d256a5a7a948c336
Aug 12 14:25:15 dist-test-slave-8w9v krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544715, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:15.981817 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:15.971386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:42629 (local address 127.5.148.62:37885)
0812 14:25:15.971467 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:15.971470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:15.971747 (+ 277us) server_negotiation.cc:407] Connection header received
0812 14:25:15.972534 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:15.972535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:15.972549 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:15.972601 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:15.973685 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:15.974183 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:15.975704 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:15.975901 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:15.975961 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:15.978561 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:15.978563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:15.978563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:15.978573 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:15.980266 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:15.980710 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:15.980711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:15.980712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:15.980783 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:15.981075 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:15.981076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:15.981076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:15.981228 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:15.981309 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:15.981597 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 14:25:15.981692 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 14:25:15.981920 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:15.971366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:42629)
0812 14:25:15.971568 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:15.971608 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:15.972417 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:15.972675 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:15.972679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:15.973385 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:15.973584 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:15.973589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:15.974288 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:15.974289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:15.975610 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:15.975614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:15.976017 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:15.976018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:15.976197 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:15.976740 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:15.976751 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:15.978458 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:15.980365 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:15.980372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:15.980374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:15.980635 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:15.980900 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:15.980901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:15.980901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:15.981006 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:15.981300 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:15.981305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:15.981437 (+ 132us) client_negotiation.cc:769] Sending connection context
0812 14:25:15.981522 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 14:25:15.981693 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
I20260812 14:25:15.982824 22276 heartbeater.cc:347] Connected to a master server at 127.5.148.62:37885
I20260812 14:25:15.982941 22276 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:15.983147 22276 heartbeater.cc:511] Master 127.5.148.62:37885 requested a full tablet report, sending...
I20260812 14:25:15.983574 21899 ts_manager.cc:194] Registered new tserver with Master: b8252fe49ffc4e34b1c65dde56faf03d (127.5.148.2:39335)
I20260812 14:25:15.984001 21899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:42629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:25:16.002677 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:37885
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:16.089079 22283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:16.089294 22283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:16.089320 22283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:16.090538 22283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:16.090584 22283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:16.090612 22283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:25:16.092216 22283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:16.092480 22283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:16.092734 22283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:16.098291 22293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:16.098382 22290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:16.098287 22291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:16.098467 22283 server_base.cc:1034] running on GCE node
I20260812 14:25:16.098830 22283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:16.099148 22283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:16.100319 22283 hybrid_clock.cc:648] HybridClock initialized: now 1786544716100309 us; error 30 us; skew 500 ppm
Aug 12 14:25:16 dist-test-slave-8w9v krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:16.103001 22283 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:25:16.103538 22283 webserver.cc:460] Webserver started at http://127.5.148.3:39933/ using document root <none> and password file <none>
I20260812 14:25:16.103803 22283 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:16.103874 22283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:16.104044 22283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:25:16.105217 22283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "8b981f37acde4890a578484024528e73"
format_stamp: "Formatted at 2026-08-12 14:25:16 on dist-test-slave-8w9v"
server_key: "e78edd467e08a9f2f9a4f9e5c318095e"
server_key_iv: "8791cf0b2b8c7a7ab5b92dbefc2dc36e"
server_key_version: "encryptionkey@0"
I20260812 14:25:16.105604 22283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "8b981f37acde4890a578484024528e73"
format_stamp: "Formatted at 2026-08-12 14:25:16 on dist-test-slave-8w9v"
server_key: "e78edd467e08a9f2f9a4f9e5c318095e"
server_key_iv: "8791cf0b2b8c7a7ab5b92dbefc2dc36e"
server_key_version: "encryptionkey@0"
I20260812 14:25:16.107019 22283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:25:16.107765 22300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:16.107949 22283 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:25:16.108062 22283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "8b981f37acde4890a578484024528e73"
format_stamp: "Formatted at 2026-08-12 14:25:16 on dist-test-slave-8w9v"
server_key: "e78edd467e08a9f2f9a4f9e5c318095e"
server_key_iv: "8791cf0b2b8c7a7ab5b92dbefc2dc36e"
server_key_version: "encryptionkey@0"
I20260812 14:25:16.108182 22283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:16.120110 22283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:16.122100 22283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:16.122262 22283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:16.122485 22283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:16.122828 22283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:16.122884 22283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:16.122925 22283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:16.122963 22283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:16.129277 22283 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:35733
I20260812 14:25:16.129330 22436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:35733 every 8 connection(s)
I20260812 14:25:16.129648 22283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:25:16 dist-test-slave-8w9v krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:16.138993 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 22283
I20260812 14:25:16.139096 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:25:16.139273 5712 external_mini_cluster.cc:1428] Setting key cda4f76c542283d8d38ed3cfe9322374
I20260812 14:25:16.139407 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.130367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:45065 (local address 127.5.148.62:37885)
0812 14:25:16.130446 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.130448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.130702 (+ 254us) server_negotiation.cc:407] Connection header received
0812 14:25:16.131211 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.131212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.131221 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.131281 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:16.132464 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.133248 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.134247 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.134427 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.134479 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.136771 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:16.136772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:16.136772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:16.136779 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:16.138055 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:16.138417 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:16.138417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:16.138418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:16.138468 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:16.138685 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:16.138685 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:16.138686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:16.138785 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:16.138835 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.139221 (+ 386us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.139291 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 14:25:16.139632 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.130344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:45065)
0812 14:25:16.130537 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:16.130581 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:16.131129 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:16.131336 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:16.131340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:16.131978 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:16.132347 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.132355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.133338 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.133339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.134165 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.134167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.134504 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.134505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.134608 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.135195 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:16.135203 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:16.136686 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:16.138128 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:16.138135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:16.138137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:16.138350 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:16.138527 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:16.138528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:16.138528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:16.138630 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:16.138857 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:16.138863 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:16.139052 (+ 189us) client_negotiation.cc:769] Sending connection context
0812 14:25:16.139188 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 14:25:16.139372 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":118,"threads_started":1}
I20260812 14:25:16.140172 22437 heartbeater.cc:347] Connected to a master server at 127.5.148.62:37885
I20260812 14:25:16.140285 22437 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:16.140481 22437 heartbeater.cc:511] Master 127.5.148.62:37885 requested a full tablet report, sending...
I20260812 14:25:16.140877 21899 ts_manager.cc:194] Registered new tserver with Master: 8b981f37acde4890a578484024528e73 (127.5.148.3:35733)
I20260812 14:25:16.141218 21899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:45065
I20260812 14:25:16.151341 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:25:16.158396 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.152727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55860 (local address 127.5.148.62:37885)
0812 14:25:16.152771 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.152773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.152894 (+ 121us) server_negotiation.cc:407] Connection header received
0812 14:25:16.152961 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.152961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.152970 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.153023 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:16.153588 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.154067 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.154770 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.154957 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.155005 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.156232 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:16.156233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:16.156233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:16.156240 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:16.157424 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:16.157735 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:16.157736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:16.157736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:16.157787 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:16.157969 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:16.157970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:16.157970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:16.158110 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:16.158187 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.158275 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.158332 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:25:16.160178 21899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:25:16.161813 21899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:25:16.168645 22456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.164356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.5.148.1:46651)
0812 14:25:16.164834 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.164837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.164846 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:25:16.164882 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.164885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.165020 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.165380 (+ 360us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:16.165463 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.166493 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.167586 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.168186 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.168278 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.168345 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.168503 (+ 158us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.168545 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":118,"threads_started":1}
I20260812 14:25:16.168653 22454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.164249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:46651 (local address 127.0.0.1:59484)
0812 14:25:16.164740 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:16.164749 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:16.164784 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:16.165250 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:16.165251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:16.165253 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:16.165295 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.165297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.166575 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.166576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.167501 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.167503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.168279 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.168280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.168361 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.168374 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 14:25:16.168483 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 14:25:16.168541 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":48,"threads_started":1}
I20260812 14:25:16.168834 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.164529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.5.148.3:35733)
0812 14:25:16.164731 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.164737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.164903 (+ 166us) server_negotiation.cc:407] Connection header received
0812 14:25:16.164978 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.164981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.165019 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.165095 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:16.165306 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.166037 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.167483 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.168230 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.168301 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.168357 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.168701 (+ 344us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.168753 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260812 14:25:16.168818 22452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.164247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:39335 (local address 127.0.0.1:55314)
0812 14:25:16.164531 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:16.164563 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:16.164618 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:16.164985 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:16.164988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:16.164991 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:16.165084 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.165086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.166517 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.166518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.167659 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.167661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.168616 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.168617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.168675 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.168678 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:25:16.168731 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 14:25:16.168763 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":93,"threads_started":1}
I20260812 14:25:16.168660 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.164247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:35733 (local address 127.0.0.1:60502)
0812 14:25:16.164844 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:16.164850 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:16.164905 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:16.165184 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:16.165185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:16.165187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:16.165235 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.165237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.166155 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.166156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.167390 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.167393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.168393 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.168394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.168468 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.168472 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:25:16.168527 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 14:25:16.168569 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":4,"thread_start_us":96,"threads_started":1}
I20260812 14:25:16.169155 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.164612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55314 (local address 127.5.148.2:39335)
0812 14:25:16.164776 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.164781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.164794 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:25:16.164839 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.164841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.164859 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.165168 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:16.165251 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.166402 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.167748 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.168471 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.168534 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.168583 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.169015 (+ 432us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.169073 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260812 14:25:16.169806 22356 tablet_service.cc:1467] Processing CreateTablet for tablet d04c40e468eb4c8a9a86fa3c38b88308 (DEFAULT_TABLE table=test-table [id=c1dabb4f704c4f5daba73348e277627f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:25:16.169857 22041 tablet_service.cc:1467] Processing CreateTablet for tablet d04c40e468eb4c8a9a86fa3c38b88308 (DEFAULT_TABLE table=test-table [id=c1dabb4f704c4f5daba73348e277627f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:25:16.169857 22205 tablet_service.cc:1467] Processing CreateTablet for tablet d04c40e468eb4c8a9a86fa3c38b88308 (DEFAULT_TABLE table=test-table [id=c1dabb4f704c4f5daba73348e277627f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:25:16.170171 22356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d04c40e468eb4c8a9a86fa3c38b88308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:16.170198 22041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d04c40e468eb4c8a9a86fa3c38b88308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:16.170198 22205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d04c40e468eb4c8a9a86fa3c38b88308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:16.172742 22461 tablet_bootstrap.cc:492] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73: Bootstrap starting.
I20260812 14:25:16.173444 22461 tablet_bootstrap.cc:654] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:16.173597 22463 tablet_bootstrap.cc:492] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d: Bootstrap starting.
I20260812 14:25:16.174283 22461 log.cc:824] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:16.174352 22463 tablet_bootstrap.cc:654] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:16.174903 22463 log.cc:824] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:16.175132 22461 tablet_bootstrap.cc:492] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73: No bootstrap required, opened a new log
I20260812 14:25:16.175208 22461 ts_tablet_manager.cc:1397] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:25:16.175268 22462 tablet_bootstrap.cc:492] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: Bootstrap starting.
I20260812 14:25:16.175614 22463 tablet_bootstrap.cc:492] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d: No bootstrap required, opened a new log
I20260812 14:25:16.175699 22463 ts_tablet_manager.cc:1397] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:25:16.175904 22462 tablet_bootstrap.cc:654] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:16.176559 22462 log.cc:824] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:16.177469 22462 tablet_bootstrap.cc:492] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: No bootstrap required, opened a new log
I20260812 14:25:16.177538 22462 ts_tablet_manager.cc:1397] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:25:16.177429 22461 raft_consensus.cc:348] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } } peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } }
I20260812 14:25:16.177636 22461 raft_consensus.cc:374] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:16.177708 22461 raft_consensus.cc:729] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b981f37acde4890a578484024528e73, State: Initialized, Role: FOLLOWER
I20260812 14:25:16.177661 22463 raft_consensus.cc:348] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } } peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } }
I20260812 14:25:16.177840 22463 raft_consensus.cc:374] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:16.177855 22461 consensus_queue.cc:260] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } } peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } }
I20260812 14:25:16.177911 22463 raft_consensus.cc:729] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8252fe49ffc4e34b1c65dde56faf03d, State: Initialized, Role: FOLLOWER
I20260812 14:25:16.178049 22463 consensus_queue.cc:260] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } } peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } }
I20260812 14:25:16.178164 22461 ts_tablet_manager.cc:1428] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:25:16.178176 22437 heartbeater.cc:503] Master 127.5.148.62:37885 was elected leader, sending a full tablet report...
I20260812 14:25:16.178457 22276 heartbeater.cc:503] Master 127.5.148.62:37885 was elected leader, sending a full tablet report...
I20260812 14:25:16.178463 22463 ts_tablet_manager.cc:1428] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:25:16.179672 22462 raft_consensus.cc:348] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } } peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } }
I20260812 14:25:16.179828 22462 raft_consensus.cc:374] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:16.179872 22462 raft_consensus.cc:729] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5966afdbdc46e8b5738cf0218cbcc4, State: Initialized, Role: FOLLOWER
I20260812 14:25:16.179979 22462 consensus_queue.cc:260] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } } peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } }
I20260812 14:25:16.180281 22462 ts_tablet_manager.cc:1428] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 14:25:16.180346 22117 heartbeater.cc:503] Master 127.5.148.62:37885 was elected leader, sending a full tablet report...
W20260812 14:25:16.221230 22277 tablet.cc:2369] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:25:16.255553 22468 raft_consensus.cc:482] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:25:16.255700 22468 raft_consensus.cc:504] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } } peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } }
I20260812 14:25:16.256316 22468 leader_election.cc:288] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8252fe49ffc4e34b1c65dde56faf03d (127.5.148.2:39335), 2d5966afdbdc46e8b5738cf0218cbcc4 (127.5.148.1:46651)
I20260812 14:25:16.259519 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.256676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:58879 (local address 127.5.148.2:39335)
0812 14:25:16.256722 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.256725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.256870 (+ 145us) server_negotiation.cc:407] Connection header received
0812 14:25:16.256961 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.256962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.256976 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.257041 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:16.257236 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.257691 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.258757 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.259144 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.259235 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.259247 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.259398 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.259437 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:25:16.259552 22456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.256680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:33699 (local address 127.5.148.1:46651)
0812 14:25:16.256729 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.256731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.256744 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:25:16.256851 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.256852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.256867 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.256933 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:16.257121 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.257686 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.258706 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.259172 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.259256 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.259267 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.259440 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.259480 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:25:16.259582 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.256563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:46651 (local address 127.5.148.3:33699)
0812 14:25:16.256647 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:16.256664 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:16.256735 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:16.256949 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:16.256951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:16.256953 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:16.257041 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.257043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.257779 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.257781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.258590 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.258593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.259256 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.259257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.259345 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.259348 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:25:16.259425 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 14:25:16.259466 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":3}
I20260812 14:25:16.259608 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.256563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:39335 (local address 127.5.148.3:58879)
0812 14:25:16.256781 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:16.256790 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:16.256883 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:16.257056 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:16.257057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:16.257058 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:16.257160 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.257162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.257785 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.257786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.258616 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.258619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.259239 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.259240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.259320 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.259323 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:25:16.259391 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 14:25:16.259435 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":23,"thread_start_us":130,"threads_started":1}
I20260812 14:25:16.260175 22228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04c40e468eb4c8a9a86fa3c38b88308" candidate_uuid: "8b981f37acde4890a578484024528e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" is_pre_election: true
I20260812 14:25:16.260375 22228 raft_consensus.cc:2454] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b981f37acde4890a578484024528e73 in term 0.
I20260812 14:25:16.260414 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04c40e468eb4c8a9a86fa3c38b88308" candidate_uuid: "8b981f37acde4890a578484024528e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" is_pre_election: true
I20260812 14:25:16.260556 22063 raft_consensus.cc:2454] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b981f37acde4890a578484024528e73 in term 0.
I20260812 14:25:16.260653 22303 leader_election.cc:302] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b981f37acde4890a578484024528e73, b8252fe49ffc4e34b1c65dde56faf03d; no voters:
I20260812 14:25:16.260818 22468 raft_consensus.cc:2781] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:25:16.260890 22468 raft_consensus.cc:482] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:25:16.260926 22468 raft_consensus.cc:3037] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:16.261678 22468 raft_consensus.cc:504] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } } peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } }
I20260812 14:25:16.261834 22468 leader_election.cc:288] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [CANDIDATE]: Term 1 election: Requested vote from peers b8252fe49ffc4e34b1c65dde56faf03d (127.5.148.2:39335), 2d5966afdbdc46e8b5738cf0218cbcc4 (127.5.148.1:46651)
I20260812 14:25:16.262012 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04c40e468eb4c8a9a86fa3c38b88308" candidate_uuid: "8b981f37acde4890a578484024528e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4"
I20260812 14:25:16.262024 22228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04c40e468eb4c8a9a86fa3c38b88308" candidate_uuid: "8b981f37acde4890a578484024528e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8252fe49ffc4e34b1c65dde56faf03d"
I20260812 14:25:16.262110 22063 raft_consensus.cc:3037] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:16.262110 22228 raft_consensus.cc:3037] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:16.262852 22063 raft_consensus.cc:2454] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b981f37acde4890a578484024528e73 in term 1.
I20260812 14:25:16.262868 22228 raft_consensus.cc:2454] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b981f37acde4890a578484024528e73 in term 1.
I20260812 14:25:16.263057 22303 leader_election.cc:302] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b981f37acde4890a578484024528e73, b8252fe49ffc4e34b1c65dde56faf03d; no voters:
I20260812 14:25:16.263260 22468 raft_consensus.cc:2781] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:25:16.263494 22468 raft_consensus.cc:686] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [term 1 LEADER]: Becoming Leader. State: Replica: 8b981f37acde4890a578484024528e73, State: Running, Role: LEADER
I20260812 14:25:16.263615 22468 consensus_queue.cc:237] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } } peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } }
I20260812 14:25:16.264546 21898 catalog_manager.cc:5495] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b981f37acde4890a578484024528e73 (127.5.148.3). New cstate: current_term: 1 leader_uuid: "8b981f37acde4890a578484024528e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } health_report { overall_health: HEALTHY } } }
I20260812 14:25:16.288766 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.286201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.5.148.3:35733)
0812 14:25:16.286268 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.286271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.286283 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:25:16.286333 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.286333 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.286368 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.286446 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:25:16.286604 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.287118 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.287886 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.288085 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.288234 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.288418 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:25:16.288533 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:25:16.288633 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.288652 (+ 19us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.288674 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 14:25:16.290671 22063 raft_consensus.cc:1261] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Refusing update from remote peer 8b981f37acde4890a578484024528e73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:25:16.290678 22228 raft_consensus.cc:1261] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d [term 1 FOLLOWER]: Refusing update from remote peer 8b981f37acde4890a578484024528e73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:25:16.291040 22468 consensus_queue.cc:1035] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:25:16.291221 22473 consensus_queue.cc:1035] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:25:16.293666 22476 mvcc.cc:205] Tried to move back new op lower bound from 7317687157923782656 to 7317687157816578048. Current Snapshot: MvccSnapshot[applied={T|T < 7317687157923782656}]
I20260812 14:25:16.294646 22478 mvcc.cc:205] Tried to move back new op lower bound from 7317687157923782656 to 7317687157816578048. Current Snapshot: MvccSnapshot[applied={T|T < 7317687157923782656 or (T in {7317687157923782656})}]
I20260812 14:25:16.295039 22477 mvcc.cc:205] Tried to move back new op lower bound from 7317687157923782656 to 7317687157816578048. Current Snapshot: MvccSnapshot[applied={T|T < 7317687157923782656}]
I20260812 14:25:16.296039 21898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:25:16.296172 21898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:25:16.297055 21898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: Sending DeleteTablet for 3 replicas of tablet d04c40e468eb4c8a9a86fa3c38b88308
I20260812 14:25:16.297478 22356 tablet_service.cc:1514] Processing DeleteTablet for tablet d04c40e468eb4c8a9a86fa3c38b88308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:16 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:60502
I20260812 14:25:16.297487 22205 tablet_service.cc:1514] Processing DeleteTablet for tablet d04c40e468eb4c8a9a86fa3c38b88308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:16 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:55314
I20260812 14:25:16.297487 22041 tablet_service.cc:1514] Processing DeleteTablet for tablet d04c40e468eb4c8a9a86fa3c38b88308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:16 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:59484
I20260812 14:25:16.297546 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 21970
I20260812 14:25:16.297796 22487 tablet_replica.cc:330] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d: stopping tablet replica
I20260812 14:25:16.297930 22487 raft_consensus.cc:2229] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:25:16.298970 22487 raft_consensus.cc:2258] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:25:16.299731 22487 ts_tablet_manager.cc:1905] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:25:16.301636 22487 ts_tablet_manager.cc:1918] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:25:16.301780 22487 log.cc:1196] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/d04c40e468eb4c8a9a86fa3c38b88308
I20260812 14:25:16.302167 22487 ts_tablet_manager.cc:1939] T d04c40e468eb4c8a9a86fa3c38b88308 P b8252fe49ffc4e34b1c65dde56faf03d: Deleting consensus metadata
I20260812 14:25:16.302706 21880 catalog_manager.cc:4841] TS b8252fe49ffc4e34b1c65dde56faf03d (127.5.148.2:39335): tablet d04c40e468eb4c8a9a86fa3c38b88308 (table test-table [id=c1dabb4f704c4f5daba73348e277627f]) successfully deleted
I20260812 14:25:16.302794 22486 tablet_replica.cc:330] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73: stopping tablet replica
I20260812 14:25:16.302937 22486 raft_consensus.cc:2229] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:25:16.303085 22486 raft_consensus.cc:2258] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:25:16.303783 22486 ts_tablet_manager.cc:1905] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:25:16.306202 22486 ts_tablet_manager.cc:1918] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:25:16.306311 22486 log.cc:1196] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/d04c40e468eb4c8a9a86fa3c38b88308
I20260812 14:25:16.306706 22486 ts_tablet_manager.cc:1939] T d04c40e468eb4c8a9a86fa3c38b88308 P 8b981f37acde4890a578484024528e73: Deleting consensus metadata
I20260812 14:25:16.307296 21879 catalog_manager.cc:4841] TS 8b981f37acde4890a578484024528e73 (127.5.148.3:35733): tablet d04c40e468eb4c8a9a86fa3c38b88308 (table test-table [id=c1dabb4f704c4f5daba73348e277627f]) successfully deleted
W20260812 14:25:16.313740 21882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:46651 (error 108)
I20260812 14:25:16.313910 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 22127
I20260812 14:25:16.314333 22452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.314209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:46651 (local address 127.0.0.1:59488)
0812 14:25:16.314261 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:16.314296 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:46651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260812 14:25:16.314560 21882 catalog_manager.cc:4584] TS 2d5966afdbdc46e8b5738cf0218cbcc4 (127.5.148.1:46651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d04c40e468eb4c8a9a86fa3c38b88308: Network error: Client connection negotiation failed: client connection to 127.5.148.1:46651: connect: Connection refused (error 111)
I20260812 14:25:16.321859 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 22283
I20260812 14:25:16.329823 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 21859
I20260812 14:25:16.338066 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:37885
--webserver_interface=127.5.148.62
--webserver_port=35021
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:37885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:16.422389 22489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:16.422600 22489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:16.422623 22489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:16.423695 22489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:25:16.423728 22489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:16.423743 22489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:16.423755 22489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:25:16.423767 22489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:25:16.425211 22489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:37885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:37885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=35021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:16.425503 22489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:16.425726 22489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:16.436610 22497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:16.436631 22501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:16.436614 22499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:16.436988 22489 server_base.cc:1034] running on GCE node
I20260812 14:25:16.437295 22489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:16.437711 22489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:16.438897 22489 hybrid_clock.cc:648] HybridClock initialized: now 1786544716438841 us; error 56 us; skew 500 ppm
Aug 12 14:25:16 dist-test-slave-8w9v krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:16.441798 22489 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:25:16.442232 22489 webserver.cc:460] Webserver started at http://127.5.148.62:35021/ using document root <none> and password file <none>
I20260812 14:25:16.442441 22489 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:16.442495 22489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:16.443992 22489 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:16.444713 22508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:16.444931 22489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:25:16.445031 22489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "ca6e04394f36473c94903b255ac7a270"
format_stamp: "Formatted at 2026-08-12 14:25:15 on dist-test-slave-8w9v"
server_key: "02b79de3c5a07bd30631595f0a04cb91"
server_key_iv: "874d1720ebd2192587998063808bbbcd"
server_key_version: "encryptionkey@0"
I20260812 14:25:16.445470 22489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:16.461686 22489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:16.463608 22489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:16.463721 22489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:16.468093 22489 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:37885
I20260812 14:25:16.468168 22572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:37885 every 8 connection(s)
I20260812 14:25:16.468587 22489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:25:16.470309 22573 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:25:16.471005 22573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: Bootstrap starting.
I20260812 14:25:16.473711 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 22489
I20260812 14:25:16.474026 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:46651
--local_ip_for_outbound_sockets=127.5.148.1
--tserver_master_addrs=127.5.148.62:37885
--webserver_port=40083
--webserver_interface=127.5.148.1
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:25:16.480204 22573 log.cc:824] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:16.482270 22573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:25:16.482620 22573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: Bootstrap complete.
I20260812 14:25:16.485539 22573 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } }
I20260812 14:25:16.485808 22573 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6e04394f36473c94903b255ac7a270, State: Initialized, Role: FOLLOWER
I20260812 14:25:16.486027 22573 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } }
I20260812 14:25:16.486126 22573 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:25:16.486188 22573 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:25:16.486236 22573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:25:16.487303 22573 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } }
I20260812 14:25:16.487432 22573 leader_election.cc:302] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca6e04394f36473c94903b255ac7a270; no voters:
I20260812 14:25:16.487651 22573 leader_election.cc:288] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 14:25:16.487690 22577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 14:25:16.487807 22577 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 2 LEADER]: Becoming Leader. State: Replica: ca6e04394f36473c94903b255ac7a270, State: Running, Role: LEADER
I20260812 14:25:16.487913 22577 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } }
I20260812 14:25:16.487986 22573 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:25:16.488226 22577 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca6e04394f36473c94903b255ac7a270. Latest consensus state: current_term: 2 leader_uuid: "ca6e04394f36473c94903b255ac7a270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } } }
I20260812 14:25:16.488413 22577 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:16.488224 22578 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ca6e04394f36473c94903b255ac7a270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } } }
I20260812 14:25:16.488722 22578 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:16.489889 22596 catalog_manager.cc:1259] Loaded cluster ID: 949fa5c383d24fc9b3ed9ba771483732
I20260812 14:25:16.489943 22596 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: loading cluster ID for follower catalog manager: success
W20260812 14:25:16.490145 22596 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260812 14:25:16.490190 22596 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260812 14:25:16.490298 22584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:25:16.490661 22584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:25:16.490741 22584 catalog_manager.cc:1259] Loaded cluster ID: 949fa5c383d24fc9b3ed9ba771483732
I20260812 14:25:16.490772 22584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:25:16.490929 22584 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 14:25:16.495641 22584 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 14:25:16.498381 22584 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:25:16.498620 22584 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: Loaded TSK: 0
I20260812 14:25:16.498865 22584 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:25:16.562572 22575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:16.562820 22575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:16.562867 22575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:16.564055 22575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:16.564122 22575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:16.564216 22575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:25:16.565745 22575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:46651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=40083
--tserver_master_addrs=127.5.148.62:37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:16.566035 22575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:16.566284 22575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:16.572270 22604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:16.572235 22606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:16.572315 22608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:16.572610 22575 server_base.cc:1034] running on GCE node
I20260812 14:25:16.572882 22575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:16.573167 22575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:16.574355 22575 hybrid_clock.cc:648] HybridClock initialized: now 1786544716574332 us; error 37 us; skew 500 ppm
Aug 12 14:25:16 dist-test-slave-8w9v krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:16.577661 22575 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:25:16.578258 22575 webserver.cc:460] Webserver started at http://127.5.148.1:40083/ using document root <none> and password file <none>
I20260812 14:25:16.578526 22575 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:16.578613 22575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:16.580502 22575 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:25:16.581295 22616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:16.581532 22575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:25:16.581633 22575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "2d5966afdbdc46e8b5738cf0218cbcc4"
format_stamp: "Formatted at 2026-08-12 14:25:15 on dist-test-slave-8w9v"
server_key: "309cb3e478583a4aaa2a4e190a1a88c6"
server_key_iv: "68d1ba2c874db918302e61656fc81a01"
server_key_version: "encryptionkey@0"
I20260812 14:25:16.582122 22575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:16.602963 22575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:16.605661 22575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:16.605839 22575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:16.606139 22575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:16.606673 22623 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 14:25:16.608191 22575 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 14:25:16.608254 22575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 14:25:16.608291 22575 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 14:25:16.609218 22575 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 14:25:16.609267 22575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 14:25:16.609352 22623 tablet_bootstrap.cc:492] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: Bootstrap starting.
I20260812 14:25:16.616539 22575 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:46651
I20260812 14:25:16.616854 22762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:46651 every 8 connection(s)
I20260812 14:25:16.616989 22575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:25:16.620661 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 22575
I20260812 14:25:16.620998 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:39335
--local_ip_for_outbound_sockets=127.5.148.2
--tserver_master_addrs=127.5.148.62:37885
--webserver_port=43999
--webserver_interface=127.5.148.2
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:25:16.621191 22623 log.cc:824] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:16.623982 22623 tablet_bootstrap.cc:492] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:25:16.624361 22623 tablet_bootstrap.cc:492] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: Bootstrap complete.
I20260812 14:25:16.625821 22623 ts_tablet_manager.cc:1397] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.011s
I20260812 14:25:16.627404 22623 raft_consensus.cc:348] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } } peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } }
I20260812 14:25:16.627808 22623 raft_consensus.cc:729] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5966afdbdc46e8b5738cf0218cbcc4, State: Initialized, Role: FOLLOWER
I20260812 14:25:16.627998 22623 consensus_queue.cc:260] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } } peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } }
I20260812 14:25:16.628372 22623 ts_tablet_manager.cc:1428] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
Aug 12 14:25:16 dist-test-slave-8w9v krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:16.641496 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.617836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:38375)
0812 14:25:16.618217 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:16.618285 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:16.619108 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:16.619661 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:16.619667 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:16.620158 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:16.621666 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.621676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.623516 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.623517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.624909 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.624912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.625996 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.625997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.626137 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.635802 (+ 9665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:16.635812 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:16.637425 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:16.639785 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:16.639794 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:16.639797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:16.640037 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:16.640357 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:16.640358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:16.640359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:16.640475 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:16.640949 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:16.640956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:16.641089 (+ 133us) client_negotiation.cc:769] Sending connection context
0812 14:25:16.641178 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 14:25:16.641333 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":93,"threads_started":1}
I20260812 14:25:16.642297 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.617947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:38375 (local address 127.5.148.62:37885)
0812 14:25:16.618151 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.618161 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.618440 (+ 279us) server_negotiation.cc:407] Connection header received
0812 14:25:16.619366 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.619374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.619433 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.619579 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:16.621837 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.623343 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.625030 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.625827 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.625982 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.637588 (+ 11606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:16.637594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:16.637594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:16.637610 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:16.639666 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:16.640157 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:16.640161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:16.640161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:16.640286 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:16.640552 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:16.640553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:16.640553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:16.640870 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:16.640953 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.641272 (+ 319us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.641430 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":110,"threads_started":1}
I20260812 14:25:16.643024 22763 heartbeater.cc:347] Connected to a master server at 127.5.148.62:37885
I20260812 14:25:16.643136 22763 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:16.643335 22763 heartbeater.cc:511] Master 127.5.148.62:37885 requested a full tablet report, sending...
I20260812 14:25:16.644187 22533 ts_manager.cc:194] Registered new tserver with Master: 2d5966afdbdc46e8b5738cf0218cbcc4 (127.5.148.1:46651)
I20260812 14:25:16.645406 22533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:38375
I20260812 14:25:16.646384 22763 heartbeater.cc:503] Master 127.5.148.62:37885 was elected leader, sending a full tablet report...
Aug 12 14:25:16 dist-test-slave-8w9v krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for kudu/127.5.148.1@KRBTEST.COM
I20260812 14:25:16.654300 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.644842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:46651 (local address 127.0.0.1:59492)
0812 14:25:16.645047 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:16.645093 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:16.645163 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:16.645379 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:16.645386 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:16.645724 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:16.645790 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.645793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.647791 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.647792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.648618 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.648621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.649202 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.649203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.649285 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.649823 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:16.649827 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:16.651097 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:16.653227 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:16.653234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:16.653236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:16.653439 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:16.653652 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:16.653653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:16.653653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:16.653705 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:16.654018 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:16.654024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:16.654069 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 14:25:16.654134 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 14:25:16.654229 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":134,"threads_started":1}
I20260812 14:25:16.654386 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.644898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.5.148.1:46651)
0812 14:25:16.645153 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.645157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.645169 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:25:16.645255 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.645258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.645279 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.645361 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:16.646578 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.647583 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.648701 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.649119 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.649196 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.651182 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:16.651190 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:16.651191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:16.651198 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:16.653130 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:16.653517 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:16.653521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:16.653521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:16.653586 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:16.653781 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:16.653782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:16.653783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:16.653952 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:16.654034 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.654189 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.654288 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":83,"threads_started":1}
I20260812 14:25:16.654788 22670 tablet_service.cc:1514] Processing DeleteTablet for tablet d04c40e468eb4c8a9a86fa3c38b88308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:16 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:59492
I20260812 14:25:16.655433 22772 tablet_replica.cc:330] stopping tablet replica
I20260812 14:25:16.655548 22772 raft_consensus.cc:2229] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:25:16.655640 22772 raft_consensus.cc:2258] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:25:16.655962 22772 ts_tablet_manager.cc:1905] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:25:16.658034 22772 ts_tablet_manager.cc:1918] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:25:16.658102 22772 log.cc:1196] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal/wals/d04c40e468eb4c8a9a86fa3c38b88308
I20260812 14:25:16.658336 22772 ts_tablet_manager.cc:1939] T d04c40e468eb4c8a9a86fa3c38b88308 P 2d5966afdbdc46e8b5738cf0218cbcc4: Deleting consensus metadata
I20260812 14:25:16.658881 22514 catalog_manager.cc:4841] TS 2d5966afdbdc46e8b5738cf0218cbcc4 (127.5.148.1:46651): tablet d04c40e468eb4c8a9a86fa3c38b88308 (table test-table [id=c1dabb4f704c4f5daba73348e277627f]) successfully deleted
W20260812 14:25:16.715036 22768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:16.715278 22768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:16.715318 22768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:16.716508 22768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:16.716575 22768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:16.716621 22768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:25:16.718113 22768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:39335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=43999
--tserver_master_addrs=127.5.148.62:37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:16.718396 22768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:16.718633 22768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:16.723747 22778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:16.723762 22777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:16.723730 22781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:16.724004 22768 server_base.cc:1034] running on GCE node
I20260812 14:25:16.724329 22768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:16.724615 22768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:16.725806 22768 hybrid_clock.cc:648] HybridClock initialized: now 1786544716725778 us; error 35 us; skew 500 ppm
Aug 12 14:25:16 dist-test-slave-8w9v krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:16.728719 22768 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:25:16.729115 22768 webserver.cc:460] Webserver started at http://127.5.148.2:43999/ using document root <none> and password file <none>
I20260812 14:25:16.729382 22768 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:16.729462 22768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:16.730974 22768 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:16.731640 22788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:16.731856 22768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:16.731952 22768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "b8252fe49ffc4e34b1c65dde56faf03d"
format_stamp: "Formatted at 2026-08-12 14:25:15 on dist-test-slave-8w9v"
server_key: "d4714e8e7602048df87c8f8d8362e91c"
server_key_iv: "2e79d138829b0a5523c4b8769e540c7c"
server_key_version: "encryptionkey@0"
I20260812 14:25:16.732410 22768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:16.749650 22768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:16.751704 22768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:16.751907 22768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:16.752216 22768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:16.752566 22768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:16.752624 22768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:16.752668 22768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:16.752694 22768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:16.759035 22768 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:39335
I20260812 14:25:16.759109 22920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:39335 every 8 connection(s)
I20260812 14:25:16.759506 22768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:25:16.759791 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 22768
I20260812 14:25:16.760110 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:35733
--local_ip_for_outbound_sockets=127.5.148.3
--tserver_master_addrs=127.5.148.62:37885
--webserver_port=39933
--webserver_interface=127.5.148.3
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:25:16 dist-test-slave-8w9v krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:16.770874 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.760422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:34843)
0812 14:25:16.760632 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:16.760682 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:16.761441 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:16.761743 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:16.761748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:16.762508 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:16.762704 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.762709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.763711 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.763712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.764590 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.764593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.765155 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.765156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.765270 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.765838 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:16.765854 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:16.767534 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:16.769496 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:16.769502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:16.769504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:16.769727 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:16.769957 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:16.769958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:16.769958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:16.770059 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:16.770374 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:16.770379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:16.770497 (+ 118us) client_negotiation.cc:769] Sending connection context
0812 14:25:16.770580 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 14:25:16.770704 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":103,"threads_started":1}
I20260812 14:25:16.770882 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.760719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:34843 (local address 127.5.148.62:37885)
0812 14:25:16.760764 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.760766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.760778 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:25:16.761589 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.761590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.761607 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.761668 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:16.762829 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.763612 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.764692 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.765048 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.765128 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.767638 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:16.767640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:16.767641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:16.767650 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:16.769406 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:16.769807 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:16.769808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:16.769809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:16.769878 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:16.770131 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:16.770132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:16.770133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:16.770295 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:16.770380 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.770633 (+ 253us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.770743 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:25:16.771469 22921 heartbeater.cc:347] Connected to a master server at 127.5.148.62:37885
I20260812 14:25:16.771566 22921 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:16.771760 22921 heartbeater.cc:511] Master 127.5.148.62:37885 requested a full tablet report, sending...
I20260812 14:25:16.772229 22533 ts_manager.cc:194] Registered new tserver with Master: b8252fe49ffc4e34b1c65dde56faf03d (127.5.148.2:39335)
I20260812 14:25:16.772615 22533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:34843
W20260812 14:25:16.847995 22924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:16.848304 22924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:16.848351 22924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:16.849651 22924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:16.849774 22924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:16.849826 22924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:25:16.851473 22924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:35733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=39933
--tserver_master_addrs=127.5.148.62:37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:16.851758 22924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:16.852005 22924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:16.857509 22933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:16.857563 22930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:16.857698 22924 server_base.cc:1034] running on GCE node
W20260812 14:25:16.857611 22931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:16.858242 22924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:16.858489 22924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:16.859634 22924 hybrid_clock.cc:648] HybridClock initialized: now 1786544716859614 us; error 43 us; skew 500 ppm
Aug 12 14:25:16 dist-test-slave-8w9v krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:16.862205 22924 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:25:16.862681 22924 webserver.cc:460] Webserver started at http://127.5.148.3:39933/ using document root <none> and password file <none>
I20260812 14:25:16.862926 22924 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:16.863013 22924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:16.864449 22924 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:25:16.865060 22941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:16.865235 22924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:25:16.865367 22924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "8b981f37acde4890a578484024528e73"
format_stamp: "Formatted at 2026-08-12 14:25:16 on dist-test-slave-8w9v"
server_key: "e78edd467e08a9f2f9a4f9e5c318095e"
server_key_iv: "8791cf0b2b8c7a7ab5b92dbefc2dc36e"
server_key_version: "encryptionkey@0"
I20260812 14:25:16.865748 22924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:16.874948 22924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:16.876953 22924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:16.877112 22924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:16.877336 22924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:16.877658 22924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:16.877718 22924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:16.877761 22924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:16.877787 22924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:16.883738 22924 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:35733
I20260812 14:25:16.883813 23072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:35733 every 8 connection(s)
I20260812 14:25:16.884238 22924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:25:16.886795 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 22924
Aug 12 14:25:16 dist-test-slave-8w9v krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:16.894423 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.887220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.5.148.62:37885)
0812 14:25:16.887350 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.887352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.887361 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:25:16.887408 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.887409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.887421 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.887492 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:16.888170 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.888665 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.889611 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.889896 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.889964 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.891248 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:16.891250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:16.891250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:16.891259 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:16.893255 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:16.893617 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:16.893618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:16.893618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:16.893679 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:16.893917 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:16.893918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:16.893918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:16.894030 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:16.894219 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.894261 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.894337 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260812 14:25:16.895346 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.885086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:42143 (local address 127.5.148.62:37885)
0812 14:25:16.885174 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.885177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.885331 (+ 154us) server_negotiation.cc:407] Connection header received
0812 14:25:16.885907 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.885909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.885920 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.885974 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:16.887097 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.887733 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.888955 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.889173 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.889225 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.892061 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:16.892063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:16.892064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:16.892072 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:16.893619 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:16.894138 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:16.894139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:16.894139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:16.894195 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:16.894587 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:16.894588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:16.894589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:16.894725 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:16.894791 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.895128 (+ 337us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.895212 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 14:25:16.895520 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.885015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:42143)
0812 14:25:16.885185 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:16.885234 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:16.885815 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:16.886036 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:16.886041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:16.886690 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:16.886990 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.887010 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.887837 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.887838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.888825 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.888828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.889271 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.889272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.889432 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.890071 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:16.890084 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:16.891971 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:16.893700 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:16.893706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:16.893709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:16.894044 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:16.894339 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:16.894340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:16.894341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:16.894489 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:16.894786 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:16.894793 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:16.894973 (+ 180us) client_negotiation.cc:769] Sending connection context
0812 14:25:16.895085 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 14:25:16.895269 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":95,"threads_started":1}
I20260812 14:25:16.895958 23073 heartbeater.cc:347] Connected to a master server at 127.5.148.62:37885
I20260812 14:25:16.896078 23073 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:16.896296 23073 heartbeater.cc:511] Master 127.5.148.62:37885 requested a full tablet report, sending...
I20260812 14:25:16.896682 22533 ts_manager.cc:194] Registered new tserver with Master: 8b981f37acde4890a578484024528e73 (127.5.148.3:35733)
I20260812 14:25:16.897037 22533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:42143
I20260812 14:25:16.906052 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:25:16.908710 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.906430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.5.148.62:37885)
0812 14:25:16.906479 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.906481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.906583 (+ 102us) server_negotiation.cc:407] Connection header received
0812 14:25:16.906648 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.906649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.906683 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.906749 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:25:16.906864 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.907322 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.908028 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.908280 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.908333 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.908473 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:25:16.908569 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:25:16.908631 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.908644 (+ 13us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.908659 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:25:16.909017 22533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:25:16.910650 22533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:25:16.912809 22670 tablet_service.cc:1467] Processing CreateTablet for tablet e6fcd29af867410180645f56bddd4240 (DEFAULT_TABLE table=test-table [id=f81d8185d0bf45abbe5cbb7bccef558f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:25:16.913054 22670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fcd29af867410180645f56bddd4240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:16.914667 22623 tablet_bootstrap.cc:492] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4: Bootstrap starting.
I20260812 14:25:16.914980 22623 tablet_bootstrap.cc:654] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:16.915722 22623 tablet_bootstrap.cc:492] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4: No bootstrap required, opened a new log
I20260812 14:25:16.915756 22623 ts_tablet_manager.cc:1397] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:25:16.915926 22623 raft_consensus.cc:348] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:16.916033 22623 raft_consensus.cc:374] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:16.916077 22623 raft_consensus.cc:729] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5966afdbdc46e8b5738cf0218cbcc4, State: Initialized, Role: FOLLOWER
I20260812 14:25:16.916154 22623 consensus_queue.cc:260] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:16.916260 22623 ts_tablet_manager.cc:1428] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:25:16.917029 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.912572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:39335 (local address 127.0.0.1:55320)
0812 14:25:16.912747 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:16.912755 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:16.912800 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:16.913209 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:16.913210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:16.913212 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:16.913251 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.913252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.914525 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.914526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.915721 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.915723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.916753 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.916754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.916820 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.916843 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 14:25:16.916900 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 14:25:16.916949 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":125,"threads_started":1}
I20260812 14:25:16.917286 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.912840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.5.148.2:39335)
0812 14:25:16.913049 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.913054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.913065 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:25:16.913108 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.913110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.913127 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.913216 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:16.913321 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.914406 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.915819 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.916659 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.916731 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.916793 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.917158 (+ 365us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.917209 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
I20260812 14:25:16.917524 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.912516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:35733 (local address 127.0.0.1:60522)
0812 14:25:16.912563 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:16.912574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:16.912611 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:16.913033 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:16.913034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:16.913036 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:16.913085 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.913086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.914342 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.914343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.915686 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:16.915691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.917213 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.917215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:16.917305 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.917310 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:25:16.917380 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 14:25:16.917442 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 14:25:16.917922 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:16.912616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.5.148.3:35733)
0812 14:25:16.912828 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:16.912833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:16.912844 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:25:16.912885 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:16.912887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:16.912902 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:16.913162 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:16.913248 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.914236 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.915935 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:16.917008 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:16.917689 (+ 681us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:16.917785 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:16.917813 (+ 28us) server_negotiation.cc:299] Negotiation successful
0812 14:25:16.917850 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
I20260812 14:25:16.918005 22843 tablet_service.cc:1467] Processing CreateTablet for tablet e6fcd29af867410180645f56bddd4240 (DEFAULT_TABLE table=test-table [id=f81d8185d0bf45abbe5cbb7bccef558f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:25:16.918445 22843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fcd29af867410180645f56bddd4240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:16.918709 22992 tablet_service.cc:1467] Processing CreateTablet for tablet e6fcd29af867410180645f56bddd4240 (DEFAULT_TABLE table=test-table [id=f81d8185d0bf45abbe5cbb7bccef558f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:25:16.919077 22992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fcd29af867410180645f56bddd4240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:16.921665 23084 tablet_bootstrap.cc:492] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d: Bootstrap starting.
I20260812 14:25:16.922472 23084 tablet_bootstrap.cc:654] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:16.922603 23085 tablet_bootstrap.cc:492] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Bootstrap starting.
I20260812 14:25:16.923101 23084 log.cc:824] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:16.923183 23085 tablet_bootstrap.cc:654] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:16.923702 23085 log.cc:824] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:16.924067 23084 tablet_bootstrap.cc:492] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d: No bootstrap required, opened a new log
I20260812 14:25:16.924189 23084 ts_tablet_manager.cc:1397] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260812 14:25:16.924530 23085 tablet_bootstrap.cc:492] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: No bootstrap required, opened a new log
I20260812 14:25:16.924614 23085 ts_tablet_manager.cc:1397] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:25:16.926165 23085 raft_consensus.cc:348] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:16.926360 23085 raft_consensus.cc:374] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:16.926403 23085 raft_consensus.cc:729] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b981f37acde4890a578484024528e73, State: Initialized, Role: FOLLOWER
I20260812 14:25:16.926160 23084 raft_consensus.cc:348] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:16.926494 23084 raft_consensus.cc:374] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:16.926553 23084 raft_consensus.cc:729] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8252fe49ffc4e34b1c65dde56faf03d, State: Initialized, Role: FOLLOWER
I20260812 14:25:16.926544 23085 consensus_queue.cc:260] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:16.926689 23084 consensus_queue.cc:260] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:16.926877 23085 ts_tablet_manager.cc:1428] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:25:16.926918 23073 heartbeater.cc:503] Master 127.5.148.62:37885 was elected leader, sending a full tablet report...
I20260812 14:25:16.927376 22921 heartbeater.cc:503] Master 127.5.148.62:37885 was elected leader, sending a full tablet report...
I20260812 14:25:16.927372 23084 ts_tablet_manager.cc:1428] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260812 14:25:17.010233 22922 tablet.cc:2369] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:25:17.117962 22764 tablet.cc:2369] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:25:17.126111 22769 raft_consensus.cc:482] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:25:17.126258 22769 raft_consensus.cc:504] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:17.126622 22769 leader_election.cc:288] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b981f37acde4890a578484024528e73 (127.5.148.3:35733), b8252fe49ffc4e34b1c65dde56faf03d (127.5.148.2:39335)
I20260812 14:25:17.129879 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:17.126940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:54741 (local address 127.5.148.3:35733)
0812 14:25:17.127001 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:17.127003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:17.127043 (+ 40us) server_negotiation.cc:407] Connection header received
0812 14:25:17.127189 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:17.127190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:17.127205 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:17.127267 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:17.127594 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:17.128066 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:17.129084 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:17.129485 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:17.129570 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:17.129581 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:17.129761 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 14:25:17.129802 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:25:17.129904 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:17.127034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:51735 (local address 127.5.148.2:39335)
0812 14:25:17.127082 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:17.127084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:17.127214 (+ 130us) server_negotiation.cc:407] Connection header received
0812 14:25:17.127299 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:17.127300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:17.127312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:17.127368 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:17.127562 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:17.128050 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:17.129079 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:17.129468 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:17.129562 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:17.129576 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:17.129779 (+ 203us) server_negotiation.cc:299] Negotiation successful
0812 14:25:17.129832 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:25:17.129928 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:17.126830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:35733 (local address 127.5.148.1:54741)
0812 14:25:17.126970 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:17.126985 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:17.127076 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:17.127274 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:17.127276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:17.127278 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:17.127409 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:17.127411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:17.128191 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:17.128192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:17.128983 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:17.128985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:17.129581 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:17.129582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:17.129665 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:17.129686 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 14:25:17.129756 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 14:25:17.129799 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":21}
I20260812 14:25:17.130155 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:17.126988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:39335 (local address 127.5.148.1:51735)
0812 14:25:17.127145 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:17.127155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:17.127221 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:17.127380 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:17.127381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:17.127382 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:17.127478 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:17.127480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:17.128153 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:17.128154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:17.128984 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:17.128986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:17.129622 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:17.129624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:17.129682 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:17.129701 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 14:25:17.129773 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 14:25:17.129814 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":53,"thread_start_us":94,"threads_started":1}
I20260812 14:25:17.130359 23019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fcd29af867410180645f56bddd4240" candidate_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b981f37acde4890a578484024528e73" is_pre_election: true
I20260812 14:25:17.130550 23019 raft_consensus.cc:2454] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d5966afdbdc46e8b5738cf0218cbcc4 in term 0.
I20260812 14:25:17.130560 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fcd29af867410180645f56bddd4240" candidate_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" is_pre_election: true
I20260812 14:25:17.130704 22864 raft_consensus.cc:2454] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d5966afdbdc46e8b5738cf0218cbcc4 in term 0.
I20260812 14:25:17.130810 22617 leader_election.cc:302] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d5966afdbdc46e8b5738cf0218cbcc4, 8b981f37acde4890a578484024528e73; no voters:
I20260812 14:25:17.130968 22769 raft_consensus.cc:2781] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:25:17.131058 22769 raft_consensus.cc:482] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:25:17.131095 22769 raft_consensus.cc:3037] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:17.131852 22769 raft_consensus.cc:504] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:17.132014 22769 leader_election.cc:288] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [CANDIDATE]: Term 1 election: Requested vote from peers 8b981f37acde4890a578484024528e73 (127.5.148.3:35733), b8252fe49ffc4e34b1c65dde56faf03d (127.5.148.2:39335)
I20260812 14:25:17.132215 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fcd29af867410180645f56bddd4240" candidate_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8252fe49ffc4e34b1c65dde56faf03d"
I20260812 14:25:17.132289 22864 raft_consensus.cc:3037] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:17.133038 22864 raft_consensus.cc:2454] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d5966afdbdc46e8b5738cf0218cbcc4 in term 1.
I20260812 14:25:17.133167 23019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fcd29af867410180645f56bddd4240" candidate_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b981f37acde4890a578484024528e73"
I20260812 14:25:17.133225 22618 leader_election.cc:302] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d5966afdbdc46e8b5738cf0218cbcc4, b8252fe49ffc4e34b1c65dde56faf03d; no voters:
I20260812 14:25:17.133255 23019 raft_consensus.cc:3037] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:17.133337 22769 raft_consensus.cc:2781] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:25:17.133517 22769 raft_consensus.cc:686] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 LEADER]: Becoming Leader. State: Replica: 2d5966afdbdc46e8b5738cf0218cbcc4, State: Running, Role: LEADER
I20260812 14:25:17.133599 22769 consensus_queue.cc:237] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:17.134189 23019 raft_consensus.cc:2454] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d5966afdbdc46e8b5738cf0218cbcc4 in term 1.
I20260812 14:25:17.134433 22533 catalog_manager.cc:5495] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d5966afdbdc46e8b5738cf0218cbcc4 (127.5.148.1). New cstate: current_term: 1 leader_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } health_report { overall_health: HEALTHY } } }
W20260812 14:25:17.134807 23074 tablet.cc:2369] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:25:17.144220 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:17.141888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.5.148.1:46651)
0812 14:25:17.141944 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:17.141946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:17.141957 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:25:17.142045 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:17.142046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:17.142062 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:17.142124 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:25:17.142275 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:17.142757 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:17.143490 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:17.143681 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:17.143731 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:17.143885 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:25:17.143991 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:25:17.144045 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:17.144087 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 14:25:17.144105 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:25:17.145733 23019 raft_consensus.cc:1261] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73 [term 1 FOLLOWER]: Refusing update from remote peer 2d5966afdbdc46e8b5738cf0218cbcc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:25:17.145727 22864 raft_consensus.cc:1261] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [term 1 FOLLOWER]: Refusing update from remote peer 2d5966afdbdc46e8b5738cf0218cbcc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:25:17.146078 22769 consensus_queue.cc:1035] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:25:17.146246 23094 consensus_queue.cc:1035] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:25:17.148790 23099 mvcc.cc:205] Tried to move back new op lower bound from 7317687161426296832 to 7317687161379835904. Current Snapshot: MvccSnapshot[applied={T|T < 7317687161426296832}]
I20260812 14:25:17.150785 22532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:25:17.150890 22532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:25:17.151634 22532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: Sending DeleteTablet for 3 replicas of tablet e6fcd29af867410180645f56bddd4240
I20260812 14:25:17.152009 22670 tablet_service.cc:1514] Processing DeleteTablet for tablet e6fcd29af867410180645f56bddd4240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:17 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:59492
I20260812 14:25:17.152030 22843 tablet_service.cc:1514] Processing DeleteTablet for tablet e6fcd29af867410180645f56bddd4240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:17 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:55320
I20260812 14:25:17.152114 22772 tablet_replica.cc:330] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4: stopping tablet replica
I20260812 14:25:17.152215 22772 raft_consensus.cc:2229] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:25:17.152293 22772 raft_consensus.cc:2258] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:25:17.152549 23108 tablet_replica.cc:330] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d: stopping tablet replica
I20260812 14:25:17.152555 22992 tablet_service.cc:1514] Processing DeleteTablet for tablet e6fcd29af867410180645f56bddd4240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:17 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:60522
I20260812 14:25:17.152590 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 22575
I20260812 14:25:17.156435 23109 tablet_replica.cc:330] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: stopping tablet replica
I20260812 14:25:17.156554 23109 raft_consensus.cc:2229] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:25:17.156661 23109 raft_consensus.cc:2258] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:25:17.157269 23109 ts_tablet_manager.cc:1905] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:25:17.159422 23109 ts_tablet_manager.cc:1918] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:25:17.159482 23109 log.cc:1196] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/e6fcd29af867410180645f56bddd4240
I20260812 14:25:17.159696 23109 ts_tablet_manager.cc:1939] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Deleting consensus metadata
I20260812 14:25:17.160192 22509 catalog_manager.cc:4841] TS 8b981f37acde4890a578484024528e73 (127.5.148.3:35733): tablet e6fcd29af867410180645f56bddd4240 (table test-table [id=f81d8185d0bf45abbe5cbb7bccef558f]) successfully deleted
W20260812 14:25:17.162066 22514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:46651 (error 108)
I20260812 14:25:17.152654 23108 raft_consensus.cc:2229] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:25:17.162308 23108 raft_consensus.cc:2258] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 14:25:17.162851 22770 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:25:17.162537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:46651 (local address 127.0.0.1:59510)
0812 14:25:17.162596 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:17.162611 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:17.162666 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:17.162804 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:46651: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":26}
I20260812 14:25:17.162860 23108 ts_tablet_manager.cc:1905] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:25:17.163144 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 22768
W20260812 14:25:17.163007 22514 catalog_manager.cc:4584] TS 2d5966afdbdc46e8b5738cf0218cbcc4 (127.5.148.1:46651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6fcd29af867410180645f56bddd4240: Network error: Client connection negotiation failed: client connection to 127.5.148.1:46651: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 14:25:17.173746 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 22924
I20260812 14:25:17.173982 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:17.173769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:39335 (local address 127.0.0.1:55334)
0812 14:25:17.173865 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:17.173932 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.2:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260812 14:25:17.182500 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 22489
I20260812 14:25:17.190555 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:37885
--webserver_interface=127.5.148.62
--webserver_port=35021
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:37885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:17.275167 23111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:17.275388 23111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:17.275410 23111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:17.276661 23111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:25:17.276706 23111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:17.276721 23111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:17.276733 23111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:25:17.276783 23111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:25:17.278272 23111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:37885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:37885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=35021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:17.278517 23111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:17.278752 23111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:17.283787 23120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:17.283789 23117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:17.283787 23118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:17.283962 23111 server_base.cc:1034] running on GCE node
I20260812 14:25:17.284345 23111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:17.284634 23111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:17.285778 23111 hybrid_clock.cc:648] HybridClock initialized: now 1786544717285764 us; error 29 us; skew 500 ppm
Aug 12 14:25:17 dist-test-slave-8w9v krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544717, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:17.292888 23111 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:25:17.293336 23111 webserver.cc:460] Webserver started at http://127.5.148.62:35021/ using document root <none> and password file <none>
I20260812 14:25:17.293569 23111 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:17.293614 23111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:17.294971 23111 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:17.295677 23127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:17.295907 23111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:17.296006 23111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "ca6e04394f36473c94903b255ac7a270"
format_stamp: "Formatted at 2026-08-12 14:25:15 on dist-test-slave-8w9v"
server_key: "02b79de3c5a07bd30631595f0a04cb91"
server_key_iv: "874d1720ebd2192587998063808bbbcd"
server_key_version: "encryptionkey@0"
I20260812 14:25:17.296440 23111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:17.313645 23111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:17.315840 23111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:17.315969 23111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:17.320580 23111 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:37885
I20260812 14:25:17.320636 23190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:37885 every 8 connection(s)
I20260812 14:25:17.321014 23111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:25:17.322659 23192 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:25:17.323350 23192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: Bootstrap starting.
I20260812 14:25:17.326329 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 23111
I20260812 14:25:17.326668 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:46651
--local_ip_for_outbound_sockets=127.5.148.1
--tserver_master_addrs=127.5.148.62:37885
--webserver_port=40083
--webserver_interface=127.5.148.1
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:25:17.332860 23192 log.cc:824] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:17.335144 23192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:25:17.335480 23192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270: Bootstrap complete.
I20260812 14:25:17.337894 23192 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } }
I20260812 14:25:17.338119 23192 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6e04394f36473c94903b255ac7a270, State: Initialized, Role: FOLLOWER
I20260812 14:25:17.338294 23192 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } }
I20260812 14:25:17.338356 23192 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:25:17.338424 23192 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:25:17.338469 23192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 2 FOLLOWER]: Advancing to term 3
I20260812 14:25:17.339538 23192 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } }
I20260812 14:25:17.339653 23192 leader_election.cc:302] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca6e04394f36473c94903b255ac7a270; no voters:
I20260812 14:25:17.339872 23192 leader_election.cc:288] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 14:25:17.339917 23197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 14:25:17.340037 23197 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [term 3 LEADER]: Becoming Leader. State: Replica: ca6e04394f36473c94903b255ac7a270, State: Running, Role: LEADER
I20260812 14:25:17.340178 23192 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:25:17.340161 23197 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } }
I20260812 14:25:17.340447 23198 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ca6e04394f36473c94903b255ac7a270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } } }
I20260812 14:25:17.340524 23198 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:17.340672 23197 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca6e04394f36473c94903b255ac7a270. Latest consensus state: current_term: 3 leader_uuid: "ca6e04394f36473c94903b255ac7a270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e04394f36473c94903b255ac7a270" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37885 } } }
I20260812 14:25:17.340929 23197 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6e04394f36473c94903b255ac7a270 [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:17.341130 23204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:25:17.342017 23204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:25:17.342140 23204 catalog_manager.cc:1259] Loaded cluster ID: 949fa5c383d24fc9b3ed9ba771483732
I20260812 14:25:17.342213 23204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 14:25:17.342700 23204 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f53bf6f39ad google::LogMessage::Fail() at ??:0
@ 0x7f53bf6f688a google::LogMessage::SendToLog() at ??:0
@ 0x7f53bf6f34de google::LogMessage::Flush() at ??:0
@ 0x7f53bf6f4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f53c76140f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f53c761d8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f53c15c10d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f53c15b6954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f53c241a6db start_thread at ??:0
@ 0x7f53be1f161f clone at ??:0
W20260812 14:25:17.449388 23194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:17.449688 23194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:17.449726 23194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:17.451560 23194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:17.451619 23194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:17.451659 23194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:25:17.453845 23194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:46651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=40083
--tserver_master_addrs=127.5.148.62:37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:17.454105 23194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:17.454376 23194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:17.460187 23234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:17.460208 23236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:17.460474 23238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:17.460647 23194 server_base.cc:1034] running on GCE node
I20260812 14:25:17.460873 23194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:17.462194 23194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:17.463418 23194 hybrid_clock.cc:648] HybridClock initialized: now 1786544717463393 us; error 37 us; skew 500 ppm
Aug 12 14:25:17 dist-test-slave-8w9v krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544717, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:17.466776 23194 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:25:17.467255 23194 webserver.cc:460] Webserver started at http://127.5.148.1:40083/ using document root <none> and password file <none>
I20260812 14:25:17.467521 23194 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:17.467599 23194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:17.469152 23194 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:25:17.470005 23245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:17.470237 23194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:25:17.470371 23194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "2d5966afdbdc46e8b5738cf0218cbcc4"
format_stamp: "Formatted at 2026-08-12 14:25:15 on dist-test-slave-8w9v"
server_key: "309cb3e478583a4aaa2a4e190a1a88c6"
server_key_iv: "68d1ba2c874db918302e61656fc81a01"
server_key_version: "encryptionkey@0"
I20260812 14:25:17.470753 23194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:17.479852 23194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:17.482295 23194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:17.482474 23194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:17.482731 23194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:17.483275 23254 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 14:25:17.485517 23194 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 14:25:17.485623 23194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260812 14:25:17.485685 23194 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 14:25:17.486480 23194 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 14:25:17.486552 23194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 14:25:17.486620 23254 tablet_bootstrap.cc:492] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4: Bootstrap starting.
I20260812 14:25:17.496114 23194 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:46651
I20260812 14:25:17.496212 23373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:46651 every 8 connection(s)
I20260812 14:25:17.496691 23194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:25:17.498416 23254 log.cc:824] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:17.501467 23254 tablet_bootstrap.cc:492] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:25:17.501832 23254 tablet_bootstrap.cc:492] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4: Bootstrap complete.
I20260812 14:25:17.503340 23254 ts_tablet_manager.cc:1397] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20260812 14:25:17.503494 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 23194
I20260812 14:25:17.503760 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:39335
--local_ip_for_outbound_sockets=127.5.148.2
--tserver_master_addrs=127.5.148.62:37885
--webserver_port=43999
--webserver_interface=127.5.148.2
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:25:17.504868 23254 raft_consensus.cc:348] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:17.505208 23254 raft_consensus.cc:729] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5966afdbdc46e8b5738cf0218cbcc4, State: Initialized, Role: FOLLOWER
I20260812 14:25:17.505385 23254 consensus_queue.cc:260] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:17.505667 23254 ts_tablet_manager.cc:1428] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4: Time spent starting tablet: real 0.002s user 0.002s sys 0.003s
W20260812 14:25:17.603463 23376 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:25:17.497672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:46689)
0812 14:25:17.497904 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:17.497962 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:17.498612 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:17.603195 (+104583us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":127,"threads_started":1}
W20260812 14:25:17.603785 23374 heartbeater.cc:644] Failed to heartbeat to 127.5.148.62:37885 (0 consecutive failures): Network error: Failed to ping master at 127.5.148.62:37885: Client connection negotiation failed: client connection to 127.5.148.62:37885: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 14:25:17.604177 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:17.604037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:54971)
0812 14:25:17.604095 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:17.604151 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260812 14:25:17.604579 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:17.604493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:34087)
0812 14:25:17.604538 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:17.604563 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 14:25:17.604710 23374 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:25:17.616264 23378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:17.616477 23378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:17.616503 23378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:17.617712 23378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:17.617759 23378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:17.617786 23378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:25:17.619288 23378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:39335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=43999
--tserver_master_addrs=127.5.148.62:37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:17.619532 23378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:17.619786 23378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:17.625329 23387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:17.625339 23386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:17.625375 23378 server_base.cc:1034] running on GCE node
W20260812 14:25:17.625344 23389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:17.625783 23378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:17.626070 23378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:17.627286 23378 hybrid_clock.cc:648] HybridClock initialized: now 1786544717627257 us; error 38 us; skew 500 ppm
Aug 12 14:25:17 dist-test-slave-8w9v krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544717, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:17.630172 23378 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:25:17.630689 23378 webserver.cc:460] Webserver started at http://127.5.148.2:43999/ using document root <none> and password file <none>
I20260812 14:25:17.630939 23378 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:17.631012 23378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:17.632517 23378 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:25:17.633311 23400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:17.633496 23378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:25:17.633603 23378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "b8252fe49ffc4e34b1c65dde56faf03d"
format_stamp: "Formatted at 2026-08-12 14:25:15 on dist-test-slave-8w9v"
server_key: "d4714e8e7602048df87c8f8d8362e91c"
server_key_iv: "2e79d138829b0a5523c4b8769e540c7c"
server_key_version: "encryptionkey@0"
I20260812 14:25:17.633977 23378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:17.647727 23378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:17.650133 23378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:17.650305 23378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:17.650555 23378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:17.651036 23408 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 14:25:17.652634 23378 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 14:25:17.652691 23378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 14:25:17.652719 23378 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 14:25:17.653398 23378 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 14:25:17.653446 23378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 14:25:17.653501 23408 tablet_bootstrap.cc:492] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d: Bootstrap starting.
I20260812 14:25:17.655648 23408 log.cc:824] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:17.659409 23408 tablet_bootstrap.cc:492] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:25:17.659758 23408 tablet_bootstrap.cc:492] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d: Bootstrap complete.
I20260812 14:25:17.659938 23408 ts_tablet_manager.cc:1397] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260812 14:25:17.661191 23378 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:39335
I20260812 14:25:17.661258 23540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:39335 every 8 connection(s)
I20260812 14:25:17.661577 23378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:25:17.662122 23408 raft_consensus.cc:348] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:17.662529 23408 raft_consensus.cc:729] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8252fe49ffc4e34b1c65dde56faf03d, State: Initialized, Role: FOLLOWER
I20260812 14:25:17.662685 23408 consensus_queue.cc:260] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:17.662848 23543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:17.662399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:40775)
0812 14:25:17.662602 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:17.662692 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":88,"threads_started":1}
I20260812 14:25:17.662941 23408 ts_tablet_manager.cc:1428] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
W20260812 14:25:17.663130 23541 heartbeater.cc:644] Failed to heartbeat to 127.5.148.62:37885 (0 consecutive failures): Network error: Failed to ping master at 127.5.148.62:37885: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
I20260812 14:25:17.663384 23543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:17.663300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:47541)
0812 14:25:17.663337 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:17.663365 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260812 14:25:17.663704 23543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:17.663630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:36467)
0812 14:25:17.663670 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:17.663691 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 14:25:17.663817 23541 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 14:25:17.671442 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 23378
I20260812 14:25:17.671684 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:35733
--local_ip_for_outbound_sockets=127.5.148.3
--tserver_master_addrs=127.5.148.62:37885
--webserver_port=39933
--webserver_interface=127.5.148.3
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:17.758986 23545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:17.759194 23545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:17.759218 23545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:17.760387 23545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:17.760432 23545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:17.760458 23545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:25:17.761857 23545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:35733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=39933
--tserver_master_addrs=127.5.148.62:37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:17.762060 23545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:17.762259 23545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:17.767580 23555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:17.767663 23545 server_base.cc:1034] running on GCE node
W20260812 14:25:17.767656 23553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:17.767580 23552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:17.768095 23545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:17.768373 23545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:17.769512 23545 hybrid_clock.cc:648] HybridClock initialized: now 1786544717769503 us; error 32 us; skew 500 ppm
Aug 12 14:25:17 dist-test-slave-8w9v krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544717, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:17.772239 23545 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:25:17.772747 23545 webserver.cc:460] Webserver started at http://127.5.148.3:39933/ using document root <none> and password file <none>
I20260812 14:25:17.773012 23545 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:17.773084 23545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:17.774580 23545 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:25:17.775292 23564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:17.775513 23545 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:17.775628 23545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "8b981f37acde4890a578484024528e73"
format_stamp: "Formatted at 2026-08-12 14:25:16 on dist-test-slave-8w9v"
server_key: "e78edd467e08a9f2f9a4f9e5c318095e"
server_key_iv: "8791cf0b2b8c7a7ab5b92dbefc2dc36e"
server_key_version: "encryptionkey@0"
I20260812 14:25:17.776023 23545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:17.789525 23545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:17.791497 23545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:17.791674 23545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:17.791944 23545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:17.792311 23545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:17.792369 23545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:17.792407 23545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:17.792429 23545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:17.798427 23545 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:35733
I20260812 14:25:17.798485 23697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:35733 every 8 connection(s)
I20260812 14:25:17.798830 23545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:25:17.799932 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:17.799547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:41461)
0812 14:25:17.799711 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:17.799786 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
W20260812 14:25:17.800420 23698 heartbeater.cc:644] Failed to heartbeat to 127.5.148.62:37885 (0 consecutive failures): Network error: Failed to ping master at 127.5.148.62:37885: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
I20260812 14:25:17.800737 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:17.800627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:38169)
0812 14:25:17.800689 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:17.800718 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 14:25:17.800993 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:17.800942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:43607)
0812 14:25:17.800963 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:17.800982 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260812 14:25:17.801108 23698 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 14:25:17.808388 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 23545
I20260812 14:25:18.605368 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:18.605120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:37321)
0812 14:25:18.605257 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:18.605299 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260812 14:25:18.664458 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:18.664232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:50469)
0812 14:25:18.664365 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:18.664408 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260812 14:25:18.801687 23705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:18.801459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:50331)
0812 14:25:18.801588 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:18.801630 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260812 14:25:18.833966 23706 raft_consensus.cc:482] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:25:18.834115 23706 raft_consensus.cc:504] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:18.834525 23706 leader_election.cc:288] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b981f37acde4890a578484024528e73 (127.5.148.3:35733), b8252fe49ffc4e34b1c65dde56faf03d (127.5.148.2:39335)
Aug 12 14:25:18 dist-test-slave-8w9v krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544717, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.2@KRBTEST.COM
Aug 12 14:25:18 dist-test-slave-8w9v krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544717, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.3@KRBTEST.COM
I20260812 14:25:18.845630 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:18.834685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:35733 (local address 127.5.148.1:56181)
0812 14:25:18.834725 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:18.834736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:18.834805 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:18.835842 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:18.835843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:18.836024 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:18.836154 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:18.836163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:18.837369 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:18.837371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:18.838650 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:18.838652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:18.839237 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:18.839238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:18.839354 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:18.840024 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:18.840029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:18.841830 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:18.843732 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:18.843741 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:18.843743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:18.844068 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:18.844455 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:18.844456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:18.844456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:18.844593 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:18.845145 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:18.845146 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:18.845208 (+ 62us) client_negotiation.cc:769] Sending connection context
0812 14:25:18.845313 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 14:25:18.845522 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":59}
I20260812 14:25:18.845654 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:18.834867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:39933 (local address 127.5.148.2:39335)
0812 14:25:18.835029 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:18.835039 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:18.835069 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:25:18.835173 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:18.835175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:18.835214 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:18.835319 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:18.836291 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:18.837333 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:18.838777 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:18.839172 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:18.839306 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:18.841511 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:18.841516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:18.841516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:18.841529 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:18.843859 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:18.844282 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:18.844285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:18.844285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:18.844437 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:18.844683 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:18.844684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:18.844684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:18.844977 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:18.845050 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:18.845398 (+ 348us) server_negotiation.cc:299] Negotiation successful
0812 14:25:18.845534 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260812 14:25:18.845667 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:18.834816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:56181 (local address 127.5.148.3:35733)
0812 14:25:18.835011 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:18.835024 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:18.835064 (+ 40us) server_negotiation.cc:407] Connection header received
0812 14:25:18.835657 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:18.835661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:18.835712 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:18.835879 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:18.836301 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:18.837201 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:18.838732 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:18.839139 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:18.839227 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:18.841957 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:18.841960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:18.841961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:18.841971 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:18.843626 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:18.844193 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:18.844197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:18.844197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:18.844352 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:18.844766 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:18.844767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:18.844768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:18.845076 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:18.845164 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:18.845384 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 14:25:18.845553 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260812 14:25:18.846046 23707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:18.834719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:39335 (local address 127.5.148.1:39933)
0812 14:25:18.834891 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:18.834901 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:18.834957 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:18.835432 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:18.835435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:18.835873 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:18.836164 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:18.836168 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:18.837417 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:18.837418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:18.838649 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:18.838652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:18.839293 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:18.839294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:18.839365 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:18.839862 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:18.839870 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:18.841387 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:18.843957 (+ 2570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:18.843958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:18.843958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:18.844209 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:18.844497 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:18.844498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:18.844499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:18.844602 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:18.845055 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:18.845062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:18.845215 (+ 153us) client_negotiation.cc:769] Sending connection context
0812 14:25:18.845310 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 14:25:18.845604 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":213,"thread_start_us":97,"threads_started":1}
W20260812 14:25:18.846540 23247 leader_election.cc:341] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b981f37acde4890a578484024528e73 (127.5.148.3:35733): Not found: Tablet not found: e6fcd29af867410180645f56bddd4240
I20260812 14:25:18.846532 23484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fcd29af867410180645f56bddd4240" candidate_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" is_pre_election: true
I20260812 14:25:18.846690 23484 raft_consensus.cc:2454] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d5966afdbdc46e8b5738cf0218cbcc4 in term 1.
I20260812 14:25:18.846937 23248 leader_election.cc:302] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d5966afdbdc46e8b5738cf0218cbcc4, b8252fe49ffc4e34b1c65dde56faf03d; no voters: 8b981f37acde4890a578484024528e73
I20260812 14:25:18.847071 23706 raft_consensus.cc:2781] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 14:25:18.847149 23706 raft_consensus.cc:482] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:25:18.847186 23706 raft_consensus.cc:3037] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:25:18.848424 23706 raft_consensus.cc:504] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:18.848606 23706 leader_election.cc:288] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [CANDIDATE]: Term 2 election: Requested vote from peers 8b981f37acde4890a578484024528e73 (127.5.148.3:35733), b8252fe49ffc4e34b1c65dde56faf03d (127.5.148.2:39335)
I20260812 14:25:18.848788 23484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fcd29af867410180645f56bddd4240" candidate_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b8252fe49ffc4e34b1c65dde56faf03d"
I20260812 14:25:18.848891 23484 raft_consensus.cc:3037] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [term 1 FOLLOWER]: Advancing to term 2
W20260812 14:25:18.848918 23247 leader_election.cc:341] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 8b981f37acde4890a578484024528e73 (127.5.148.3:35733): Not found: Tablet not found: e6fcd29af867410180645f56bddd4240
I20260812 14:25:18.850001 23484 raft_consensus.cc:2454] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d5966afdbdc46e8b5738cf0218cbcc4 in term 2.
I20260812 14:25:18.850172 23248 leader_election.cc:302] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d5966afdbdc46e8b5738cf0218cbcc4, b8252fe49ffc4e34b1c65dde56faf03d; no voters: 8b981f37acde4890a578484024528e73
I20260812 14:25:18.850260 23706 raft_consensus.cc:2781] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 14:25:18.850391 23706 raft_consensus.cc:686] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [term 2 LEADER]: Becoming Leader. State: Replica: 2d5966afdbdc46e8b5738cf0218cbcc4, State: Running, Role: LEADER
I20260812 14:25:18.850492 23706 consensus_queue.cc:237] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:18.850706 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:18.850611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:59707)
0812 14:25:18.850651 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:18.850684 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260812 14:25:19.242254 23247 consensus_peers.cc:487] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 -> Peer 8b981f37acde4890a578484024528e73 (127.5.148.3:35733): Couldn't send request to peer 8b981f37acde4890a578484024528e73. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6fcd29af867410180645f56bddd4240. This is attempt 1: this message will repeat every 5th retry.
I20260812 14:25:19.280213 23484 raft_consensus.cc:1261] T e6fcd29af867410180645f56bddd4240 P b8252fe49ffc4e34b1c65dde56faf03d [term 2 FOLLOWER]: Refusing update from remote peer 2d5966afdbdc46e8b5738cf0218cbcc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260812 14:25:19.280660 23706 consensus_queue.cc:1035] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 14:25:19.281780 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:19.281562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:48901)
0812 14:25:19.281684 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:19.281737 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260812 14:25:19.282711 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:19.282597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:48805)
0812 14:25:19.282644 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:19.282672 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 14:25:19.283069 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:19.282964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:42837)
0812 14:25:19.283011 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:19.283036 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 14:25:19.802358 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:19.802147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:44929)
0812 14:25:19.802266 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:19.802308 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260812 14:25:19.851918 23719 ts_tablet_manager.cc:927] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Initiating tablet copy from peer 2d5966afdbdc46e8b5738cf0218cbcc4 (127.5.148.1:46651)
I20260812 14:25:19.852236 23719 tablet_copy_client.cc:316] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: tablet copy: Beginning tablet copy session from remote peer at address 127.5.148.1:46651
Aug 12 14:25:19 dist-test-slave-8w9v krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544717, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.1@KRBTEST.COM
I20260812 14:25:19.860481 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:19.852661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:47751 (local address 127.5.148.1:46651)
0812 14:25:19.852836 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:19.852859 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:19.852873 (+ 14us) server_negotiation.cc:407] Connection header received
0812 14:25:19.852903 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:19.852905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:19.852921 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:19.853034 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:19.853505 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:19.854192 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:19.854927 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:19.855222 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:19.855279 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:19.857398 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:19.857406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:19.857407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:19.857413 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:19.859132 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:19.859563 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:19.859566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:19.859566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:19.859633 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:19.859850 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:19.859851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:19.859851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:19.860019 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:19.860101 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:19.860337 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 14:25:19.860410 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260812 14:25:19.860491 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:19.852557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:46651 (local address 127.5.148.3:47751)
0812 14:25:19.852639 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:19.852661 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:19.852745 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:19.853056 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:19.853062 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:19.853349 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:19.853428 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:19.853429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:19.854267 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:19.854267 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:19.854848 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:19.854850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:19.855290 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:19.855291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:19.855350 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:19.855839 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:19.855844 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:19.857309 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:19.859239 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:19.859247 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:19.859249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:19.859458 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:19.859708 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:19.859709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:19.859709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:19.859770 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:19.860087 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:19.860094 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:19.860209 (+ 115us) client_negotiation.cc:769] Sending connection context
0812 14:25:19.860285 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 14:25:19.860396 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260812 14:25:19.861095 23346 tablet_copy_service.cc:140] P 2d5966afdbdc46e8b5738cf0218cbcc4: Received BeginTabletCopySession request for tablet e6fcd29af867410180645f56bddd4240 from peer 8b981f37acde4890a578484024528e73 ({username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:47751)
I20260812 14:25:19.861182 23346 tablet_copy_service.cc:161] P 2d5966afdbdc46e8b5738cf0218cbcc4: Beginning new tablet copy session on tablet e6fcd29af867410180645f56bddd4240 from peer 8b981f37acde4890a578484024528e73 at {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:47751: session id = 8b981f37acde4890a578484024528e73-e6fcd29af867410180645f56bddd4240
I20260812 14:25:19.861711 23346 tablet_copy_source_session.cc:205] T e6fcd29af867410180645f56bddd4240 P 2d5966afdbdc46e8b5738cf0218cbcc4: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 14:25:19.862838 23719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fcd29af867410180645f56bddd4240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:19.865760 23719 tablet_copy_client.cc:791] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: tablet copy: Starting download of 0 data blocks...
I20260812 14:25:19.865932 23719 tablet_copy_client.cc:655] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: tablet copy: Starting download of 1 WAL segments...
I20260812 14:25:19.866868 23719 tablet_copy_client.cc:531] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 14:25:19.867614 23719 tablet_bootstrap.cc:492] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Bootstrap starting.
I20260812 14:25:19.869081 23719 log.cc:824] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:19.871614 23719 tablet_bootstrap.cc:492] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:25:19.871866 23719 tablet_bootstrap.cc:492] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Bootstrap complete.
I20260812 14:25:19.872021 23719 ts_tablet_manager.cc:1397] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260812 14:25:19.873083 23719 raft_consensus.cc:348] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:19.873265 23719 raft_consensus.cc:729] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b981f37acde4890a578484024528e73, State: Initialized, Role: FOLLOWER
I20260812 14:25:19.873414 23719 consensus_queue.cc:260] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b981f37acde4890a578484024528e73" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35733 } } peers { permanent_uuid: "b8252fe49ffc4e34b1c65dde56faf03d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39335 } } peers { permanent_uuid: "2d5966afdbdc46e8b5738cf0218cbcc4" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46651 } }
I20260812 14:25:19.873687 23719 ts_tablet_manager.cc:1428] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20260812 14:25:19.873976 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:19.873849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:48315)
0812 14:25:19.873902 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:19.873939 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 14:25:19.874027 23346 tablet_copy_service.cc:342] P 2d5966afdbdc46e8b5738cf0218cbcc4: Request end of tablet copy session 8b981f37acde4890a578484024528e73-e6fcd29af867410180645f56bddd4240 received from {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:47751
I20260812 14:25:19.874112 23346 tablet_copy_service.cc:433] P 2d5966afdbdc46e8b5738cf0218cbcc4: ending tablet copy session 8b981f37acde4890a578484024528e73-e6fcd29af867410180645f56bddd4240 on tablet e6fcd29af867410180645f56bddd4240 with peer 8b981f37acde4890a578484024528e73
I20260812 14:25:19.874330 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:19.874210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:56945)
0812 14:25:19.874270 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:19.874306 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 14:25:20.283422 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:20.283143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:37557)
0812 14:25:20.283307 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:20.283366 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":58,"threads_started":1}
I20260812 14:25:20.283699 23726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:20.283500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:54809)
0812 14:25:20.283629 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:20.283660 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":54,"threads_started":1}
I20260812 14:25:20.390427 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:20.390271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:38899)
0812 14:25:20.390332 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:20.390371 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 14:25:20.390533 23642 raft_consensus.cc:1203] T e6fcd29af867410180645f56bddd4240 P 8b981f37acde4890a578484024528e73 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260812 14:25:20.391166 23729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:20.391003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:34499)
0812 14:25:20.391100 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:20.391126 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":51,"threads_started":1}
I20260812 14:25:21.284507 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:21.284224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:34423)
0812 14:25:21.284404 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:21.284455 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
I20260812 14:25:21.391291 23731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:21.391025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:51463)
0812 14:25:21.391199 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:21.391241 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
I20260812 14:25:21.391809 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:21.391603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:35529)
0812 14:25:21.391736 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:21.391772 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260812 14:25:22.285373 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:22.285098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:40645)
0812 14:25:22.285273 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:22.285319 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":100,"threads_started":1}
I20260812 14:25:22.392165 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:22.391874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:45179)
0812 14:25:22.392026 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:22.392075 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260812 14:25:22.392503 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:22.392262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:38475)
0812 14:25:22.392429 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:22.392461 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
I20260812 14:25:23.286269 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:23.285969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:51149)
0812 14:25:23.286168 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:23.286215 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
I20260812 14:25:23.393033 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:23.392781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:38257)
0812 14:25:23.392919 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:23.392967 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260812 14:25:23.393217 23738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:23.392976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:37645)
0812 14:25:23.393106 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:23.393141 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260812 14:25:23.393512 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:23.393423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:40685)
0812 14:25:23.393458 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:23.393486 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260812 14:25:24.287213 23739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:24.286957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:41141)
0812 14:25:24.287118 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:24.287164 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260812 14:25:24.394156 23740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:24.393892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:53457)
0812 14:25:24.394063 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:24.394108 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":106,"threads_started":1}
I20260812 14:25:24.394237 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:24.394005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:45751)
0812 14:25:24.394133 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:24.394171 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260812 14:25:25.288337 23742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:25.287937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:39619)
0812 14:25:25.288184 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:25.288282 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
I20260812 14:25:25.395017 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:25.394730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:48621)
0812 14:25:25.394905 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:25.394965 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20260812 14:25:25.395077 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:25.394828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:56687)
0812 14:25:25.394995 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:25.395034 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260812 14:25:26.289183 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:26.288905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:47013)
0812 14:25:26.289076 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:26.289134 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260812 14:25:26.395881 23748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:26.395610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:36477)
0812 14:25:26.395795 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:26.395836 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":114,"threads_started":1}
I20260812 14:25:26.395880 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:26.395610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:45127)
0812 14:25:26.395776 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:26.395827 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260812 14:25:27.290025 23749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:27.289750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:47905)
0812 14:25:27.289916 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:27.289963 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20260812 14:25:27.396668 23750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:27.396408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:55923)
0812 14:25:27.396575 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:27.396617 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
I20260812 14:25:27.396683 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:27.396462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:47135)
0812 14:25:27.396595 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:27.396638 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260812 14:25:28.290966 23754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:28.290691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:55257)
0812 14:25:28.290865 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:28.290911 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260812 14:25:28.397502 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:28.397244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:56563)
0812 14:25:28.397420 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:28.397455 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
I20260812 14:25:28.397502 23756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:28.397252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:54141)
0812 14:25:28.397419 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:28.397455 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
I20260812 14:25:29.291798 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:29.291526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:39167)
0812 14:25:29.291705 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:29.291747 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":77,"threads_started":1}
I20260812 14:25:29.398299 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:29.398015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:59203)
0812 14:25:29.398183 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:29.398228 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260812 14:25:29.398288 23760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:29.398015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:57649)
0812 14:25:29.398183 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:29.398228 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260812 14:25:30.292676 23761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:30.292423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:47645)
0812 14:25:30.292581 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:30.292626 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260812 14:25:30.399055 23763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:30.398810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:33449)
0812 14:25:30.398968 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:30.399005 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260812 14:25:30.399060 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:30.398810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:49609)
0812 14:25:30.398963 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:30.399010 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260812 14:25:31.293340 23764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:31.293139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:53217)
0812 14:25:31.293257 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:31.293292 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260812 14:25:31.399914 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:31.399651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:51773)
0812 14:25:31.399824 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:31.399865 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
I20260812 14:25:31.399943 23767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:31.399651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:34069)
0812 14:25:31.399851 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:31.399884 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":104,"threads_started":1}
I20260812 14:25:32.294003 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:32.293786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.2:38867)
0812 14:25:32.293919 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:32.293952 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260812 14:25:32.400750 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:32.400446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.3:54481)
0812 14:25:32.400642 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:32.400701 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":124,"threads_started":1}
I20260812 14:25:32.400789 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:32.400446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37885 (local address 127.5.148.1:60907)
0812 14:25:32.400682 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:32.400729 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:37885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":127,"threads_started":1}
I20260812 14:25:32.810555 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 23194
I20260812 14:25:32.821478 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 23378
I20260812 14:25:32.828843 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 23545
2026-08-12T14:25:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18643 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:25:32 dist-test-slave-8w9v krb5kdc[23774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:25:32 dist-test-slave-8w9v krb5kdc[23774](info): set up 2 sockets
Aug 12 14:25:32 dist-test-slave-8w9v krb5kdc[23774](info): commencing operation
krb5kdc: starting...
W20260812 14:25:33.992214 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:25:34 dist-test-slave-8w9v krb5kdc[23774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:25:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:25:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:25:34.125526 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:33463
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:33463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:34.211043 23787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:34.211252 23787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:34.211277 23787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:34.212463 23787 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 14:25:34.212505 23787 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 14:25:34.212518 23787 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 14:25:34.212530 23787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:25:34.212544 23787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:34.212558 23787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:34.212569 23787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:25:34.212587 23787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:25:34.214049 23787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:33463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:33463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:34.214296 23787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:34.214478 23787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:34.220178 23792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:34.220212 23793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:34.220212 23795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:34.220247 23787 server_base.cc:1034] running on GCE node
I20260812 14:25:34.220952 23787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:34.221251 23787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:34.222429 23787 hybrid_clock.cc:648] HybridClock initialized: now 1786544734222425 us; error 63 us; skew 500 ppm
Aug 12 14:25:34 dist-test-slave-8w9v krb5kdc[23774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544734, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:34.225066 23787 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:25:34.225521 23787 webserver.cc:460] Webserver started at http://127.5.148.62:39057/ using document root <none> and password file <none>
I20260812 14:25:34.225780 23787 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:34.225858 23787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:34.226009 23787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:25:34.227116 23787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "6aedb4d320564078bbf24fce1b0c6f72"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "533a550a20c40400625f8d10136fddc2"
server_key_iv: "d0c094aa6bbda4b415e2bbddcac3fcfc"
server_key_version: "encryptionkey@0"
I20260812 14:25:34.227494 23787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "6aedb4d320564078bbf24fce1b0c6f72"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "533a550a20c40400625f8d10136fddc2"
server_key_iv: "d0c094aa6bbda4b415e2bbddcac3fcfc"
server_key_version: "encryptionkey@0"
I20260812 14:25:34.228952 23787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:25:34.229692 23802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:34.229866 23787 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:25:34.229959 23787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "6aedb4d320564078bbf24fce1b0c6f72"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "533a550a20c40400625f8d10136fddc2"
server_key_iv: "d0c094aa6bbda4b415e2bbddcac3fcfc"
server_key_version: "encryptionkey@0"
I20260812 14:25:34.230042 23787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:34.249826 23787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:34.252229 23787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:34.252416 23787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:34.256755 23787 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:33463
I20260812 14:25:34.256793 23863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:33463 every 8 connection(s)
I20260812 14:25:34.257217 23787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:25:34.258251 23864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:34.260855 23864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Bootstrap starting.
I20260812 14:25:34.261519 23864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:34.261595 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 23787
I20260812 14:25:34.261718 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:25:34.261942 5712 external_mini_cluster.cc:1428] Setting key 79107f200aee2e2a4875a73a3945f7e8
I20260812 14:25:34.261925 23864 log.cc:824] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:34.262902 23864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: No bootstrap required, opened a new log
I20260812 14:25:34.265632 23864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } }
I20260812 14:25:34.265820 23864 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:34.265859 23864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aedb4d320564078bbf24fce1b0c6f72, State: Initialized, Role: FOLLOWER
I20260812 14:25:34.265993 23864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } }
I20260812 14:25:34.266067 23864 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:25:34.266094 23864 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:25:34.266139 23864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:34.266798 23864 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } }
I20260812 14:25:34.266896 23864 leader_election.cc:302] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aedb4d320564078bbf24fce1b0c6f72; no voters:
I20260812 14:25:34.267114 23864 leader_election.cc:288] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:25:34.267140 23869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:25:34.267256 23869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 1 LEADER]: Becoming Leader. State: Replica: 6aedb4d320564078bbf24fce1b0c6f72, State: Running, Role: LEADER
I20260812 14:25:34.267333 23869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } }
I20260812 14:25:34.267424 23864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:25:34.267722 23869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aedb4d320564078bbf24fce1b0c6f72. Latest consensus state: current_term: 1 leader_uuid: "6aedb4d320564078bbf24fce1b0c6f72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } } }
I20260812 14:25:34.267814 23869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:34.268036 23871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aedb4d320564078bbf24fce1b0c6f72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } } }
I20260812 14:25:34.268177 23871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:34.269050 23874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 12 14:25:34 dist-test-slave-8w9v krb5kdc[23774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:34.269692 23874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:25:34.271723 23874 catalog_manager.cc:1347] Generated new cluster ID: e81b7c0d420a48f680caa4b01ca878b6
I20260812 14:25:34.271785 23874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:25:34.273945 23868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:34.262608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.5.148.62:33463)
0812 14:25:34.262803 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:34.262812 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:34.262849 (+ 37us) server_negotiation.cc:407] Connection header received
0812 14:25:34.263331 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:34.263340 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:34.263390 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:34.263517 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:34.264448 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.265357 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.266456 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.266885 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.266998 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:34.270160 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:34.270165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:34.270165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:34.270178 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:34.272252 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:34.272678 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:34.272681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:34.272682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:34.272783 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:34.273025 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:34.273025 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:34.273026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:34.273318 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:34.273499 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:34.273594 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 14:25:34.273730 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
I20260812 14:25:34.283041 23874 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:25:34.283794 23874 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:25:34.286938 23874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Generated new TSK 0
I20260812 14:25:34.287137 23874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:25:34.323553 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:33463
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:34.408763 23892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:34.408994 23892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:34.409021 23892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:34.410291 23892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:34.410351 23892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:34.410380 23892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:25:34.411998 23892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:33463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:34.412305 23892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:34.412577 23892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:34.418175 23900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:34.418217 23892 server_base.cc:1034] running on GCE node
W20260812 14:25:34.418184 23898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:34.418190 23897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:34.418668 23892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:34.418924 23892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:34.420094 23892 hybrid_clock.cc:648] HybridClock initialized: now 1786544734420078 us; error 49 us; skew 500 ppm
Aug 12 14:25:34 dist-test-slave-8w9v krb5kdc[23774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544734, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:34.422806 23892 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:25:34.423291 23892 webserver.cc:460] Webserver started at http://127.5.148.1:41985/ using document root <none> and password file <none>
I20260812 14:25:34.423558 23892 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:34.423633 23892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:34.423787 23892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:25:34.424970 23892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "fa4fbea27ce14ff58406d44e57b14293"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "5a25b2e7b8266833dbfe22ecf51c364d"
server_key_iv: "277b5388e7eee5ed1be01d4e572d15d2"
server_key_version: "encryptionkey@0"
I20260812 14:25:34.425356 23892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "fa4fbea27ce14ff58406d44e57b14293"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "5a25b2e7b8266833dbfe22ecf51c364d"
server_key_iv: "277b5388e7eee5ed1be01d4e572d15d2"
server_key_version: "encryptionkey@0"
I20260812 14:25:34.426783 23892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:25:34.427564 23910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:34.427735 23892 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:25:34.427842 23892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "fa4fbea27ce14ff58406d44e57b14293"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "5a25b2e7b8266833dbfe22ecf51c364d"
server_key_iv: "277b5388e7eee5ed1be01d4e572d15d2"
server_key_version: "encryptionkey@0"
I20260812 14:25:34.427942 23892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:34.469699 23892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:34.472105 23892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:34.472345 23892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:34.472648 23892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:34.473037 23892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:34.473106 23892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:34.473156 23892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:34.473209 23892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:34.479833 23892 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:33989
I20260812 14:25:34.479914 24043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:33989 every 8 connection(s)
I20260812 14:25:34.480382 23892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:25:34 dist-test-slave-8w9v krb5kdc[23774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544734, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:34.490572 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 23892
I20260812 14:25:34.490706 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:25:34.490793 23868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:34.481304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:46407 (local address 127.5.148.62:33463)
0812 14:25:34.481373 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:34.481375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:34.481574 (+ 199us) server_negotiation.cc:407] Connection header received
0812 14:25:34.482142 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:34.482144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:34.482156 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:34.482233 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:34.483255 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.483846 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.484971 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.485215 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.485295 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:34.487655 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:34.487656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:34.487657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:34.487664 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:34.489262 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:34.489715 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:34.489717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:34.489717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:34.489790 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:34.490054 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:34.490055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:34.490055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:34.490176 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:34.490233 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:34.490563 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 14:25:34.490673 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:25:34.490937 5712 external_mini_cluster.cc:1428] Setting key 700f98cd920c4219f1d408c6df361c67
I20260812 14:25:34.490969 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:34.481248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:46407)
0812 14:25:34.481437 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:34.481483 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:34.482005 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:34.482321 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:34.482325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:34.482949 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:34.483157 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:34.483162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.483955 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.483956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:34.484863 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:34.484866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.485311 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.485312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:34.485421 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:34.486000 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:34.486010 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:34.487541 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:34.489352 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:34.489360 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:34.489362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:34.489617 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:34.489876 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:34.489877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:34.489877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:34.489993 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:34.490236 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:34.490241 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:34.490390 (+ 149us) client_negotiation.cc:769] Sending connection context
0812 14:25:34.490503 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 14:25:34.490642 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260812 14:25:34.491720 24044 heartbeater.cc:347] Connected to a master server at 127.5.148.62:33463
I20260812 14:25:34.491842 24044 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:34.492055 24044 heartbeater.cc:511] Master 127.5.148.62:33463 requested a full tablet report, sending...
I20260812 14:25:34.492712 23822 ts_manager.cc:194] Registered new tserver with Master: fa4fbea27ce14ff58406d44e57b14293 (127.5.148.1:33989)
I20260812 14:25:34.493347 23822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:46407
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:25:34.519979 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:33463
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:34.605551 24049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:34.605779 24049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:34.605804 24049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:34.606927 24049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:34.606966 24049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:34.606992 24049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:25:34.608428 24049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:33463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:34.608623 24049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:34.608798 24049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:34.613938 24056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:34.613926 24055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:34.613926 24058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:34.614269 24049 server_base.cc:1034] running on GCE node
I20260812 14:25:34.614677 24049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:34.614928 24049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:34.616106 24049 hybrid_clock.cc:648] HybridClock initialized: now 1786544734616082 us; error 36 us; skew 500 ppm
Aug 12 14:25:34 dist-test-slave-8w9v krb5kdc[23774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544734, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:34.618701 24049 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:25:34.619210 24049 webserver.cc:460] Webserver started at http://127.5.148.2:44557/ using document root <none> and password file <none>
I20260812 14:25:34.619452 24049 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:34.619522 24049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:34.619669 24049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:25:34.620867 24049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "d20d1274fcae4bd5a72e189cc1b75f45"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "b9f60de51d52d45ef2e64b06690cc1da"
server_key_iv: "2b52d0edef6de98b1ef8309c35a5f941"
server_key_version: "encryptionkey@0"
I20260812 14:25:34.621264 24049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "d20d1274fcae4bd5a72e189cc1b75f45"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "b9f60de51d52d45ef2e64b06690cc1da"
server_key_iv: "2b52d0edef6de98b1ef8309c35a5f941"
server_key_version: "encryptionkey@0"
I20260812 14:25:34.622637 24049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:25:34.623428 24068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:34.623631 24049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:34.623737 24049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "d20d1274fcae4bd5a72e189cc1b75f45"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "b9f60de51d52d45ef2e64b06690cc1da"
server_key_iv: "2b52d0edef6de98b1ef8309c35a5f941"
server_key_version: "encryptionkey@0"
I20260812 14:25:34.623828 24049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:34.634204 24049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:34.636587 24049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:34.636760 24049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:34.637012 24049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:34.637351 24049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:34.637409 24049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:34.637446 24049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:34.637492 24049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:34.643610 24049 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:41509
I20260812 14:25:34.643656 24203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:41509 every 8 connection(s)
I20260812 14:25:34.644033 24049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:25:34.646258 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 24049
I20260812 14:25:34.646353 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:25:34.646582 5712 external_mini_cluster.cc:1428] Setting key 93dc27cf3778fe74d8cc612c4326ebf0
Aug 12 14:25:34 dist-test-slave-8w9v krb5kdc[23774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544734, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:34.654471 23868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:34.644935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:55013 (local address 127.5.148.62:33463)
0812 14:25:34.645004 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:34.645006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:34.645236 (+ 230us) server_negotiation.cc:407] Connection header received
0812 14:25:34.645719 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:34.645720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:34.645731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:34.645788 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:34.646929 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.647450 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.648573 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.648775 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.648870 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:34.651370 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:34.651372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:34.651372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:34.651381 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:34.653057 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:34.653464 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:34.653465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:34.653466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:34.653534 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:34.653776 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:34.653777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:34.653778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:34.653942 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:34.654023 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:34.654273 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 14:25:34.654361 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:25:34.654498 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:34.644908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:55013)
0812 14:25:34.645098 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:34.645145 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:34.645639 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:34.645849 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:34.645853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:34.646540 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:34.646835 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:34.646842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.647554 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.647555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:34.648480 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:34.648482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.648881 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.648882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:34.649054 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:34.649645 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:34.649658 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:34.651275 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:34.653141 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:34.653147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:34.653149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:34.653385 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:34.653604 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:34.653605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:34.653605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:34.653706 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:34.654015 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:34.654019 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:34.654135 (+ 116us) client_negotiation.cc:769] Sending connection context
0812 14:25:34.654217 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 14:25:34.654338 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
I20260812 14:25:34.655054 24204 heartbeater.cc:347] Connected to a master server at 127.5.148.62:33463
I20260812 14:25:34.655175 24204 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:34.655368 24204 heartbeater.cc:511] Master 127.5.148.62:33463 requested a full tablet report, sending...
I20260812 14:25:34.655762 23822 ts_manager.cc:194] Registered new tserver with Master: d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2:41509)
I20260812 14:25:34.656340 23822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:55013
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:25:34.678628 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:33463
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:34.764003 24210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:34.764281 24210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:34.764325 24210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:34.765552 24210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:34.765625 24210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:34.765674 24210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:25:34.767289 24210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:33463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:34.767575 24210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:34.767874 24210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:34.773191 24218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:34.773180 24216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:34.773532 24210 server_base.cc:1034] running on GCE node
W20260812 14:25:34.773166 24215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:34.773828 24210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:34.774102 24210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:34.775256 24210 hybrid_clock.cc:648] HybridClock initialized: now 1786544734775238 us; error 53 us; skew 500 ppm
Aug 12 14:25:34 dist-test-slave-8w9v krb5kdc[23774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544734, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:34.777837 24210 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:25:34.778311 24210 webserver.cc:460] Webserver started at http://127.5.148.3:40453/ using document root <none> and password file <none>
I20260812 14:25:34.778546 24210 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:34.778591 24210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:34.778759 24210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:25:34.779876 24210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "a09dfb1718a644909deaaae117d972d5"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "9762d105a8c907c115cd186b4aa2a117"
server_key_iv: "5b5ec19cc5ea48696dbfa8952f95bd25"
server_key_version: "encryptionkey@0"
I20260812 14:25:34.780272 24210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "a09dfb1718a644909deaaae117d972d5"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "9762d105a8c907c115cd186b4aa2a117"
server_key_iv: "5b5ec19cc5ea48696dbfa8952f95bd25"
server_key_version: "encryptionkey@0"
I20260812 14:25:34.781629 24210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:25:34.782366 24226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:34.782585 24210 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:25:34.782661 24210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "a09dfb1718a644909deaaae117d972d5"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "9762d105a8c907c115cd186b4aa2a117"
server_key_iv: "5b5ec19cc5ea48696dbfa8952f95bd25"
server_key_version: "encryptionkey@0"
I20260812 14:25:34.782757 24210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:34.791371 24210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:34.793298 24210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:34.793457 24210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:34.793675 24210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:34.793993 24210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:34.794051 24210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:34.794092 24210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:34.794121 24210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:34.800663 24210 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:41091
I20260812 14:25:34.800711 24369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:41091 every 8 connection(s)
I20260812 14:25:34.801029 24210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:25:34.804695 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 24210
I20260812 14:25:34.804786 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:25:34.804966 5712 external_mini_cluster.cc:1428] Setting key bd48fb2f82e32deb3fe7324160888b3d
Aug 12 14:25:34 dist-test-slave-8w9v krb5kdc[23774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544734, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:34.810966 23868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:34.801793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:49957 (local address 127.5.148.62:33463)
0812 14:25:34.801854 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:34.801856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:34.802095 (+ 239us) server_negotiation.cc:407] Connection header received
0812 14:25:34.802662 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:34.802663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:34.802678 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:34.802729 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:34.803729 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.804298 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.805559 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.805836 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.805930 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:34.808246 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:34.808248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:34.808248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:34.808255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:34.809717 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:34.810111 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:34.810112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:34.810112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:34.810164 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:34.810400 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:34.810401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:34.810401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:34.810540 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:34.810598 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:34.810819 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 14:25:34.810895 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:25:34.811223 24374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:34.801765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:49957)
0812 14:25:34.801971 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:34.802009 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:34.802541 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:34.802807 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:34.802810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:34.803410 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:34.803612 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:34.803617 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.804395 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.804396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:34.805456 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:34.805460 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.806086 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.806088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:34.806239 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:34.806726 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:34.806735 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:34.808107 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:34.809791 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:34.809797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:34.809799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:34.810025 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:34.810226 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:34.810227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:34.810227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:34.810331 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:34.810602 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:34.810606 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:34.810726 (+ 120us) client_negotiation.cc:769] Sending connection context
0812 14:25:34.810810 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 14:25:34.810938 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
I20260812 14:25:34.811857 24370 heartbeater.cc:347] Connected to a master server at 127.5.148.62:33463
I20260812 14:25:34.811973 24370 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:34.812218 24370 heartbeater.cc:511] Master 127.5.148.62:33463 requested a full tablet report, sending...
I20260812 14:25:34.812634 23822 ts_manager.cc:194] Registered new tserver with Master: a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091)
I20260812 14:25:34.813017 23822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:49957
I20260812 14:25:34.816882 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:25:34.824043 23868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:34.818356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41818 (local address 127.5.148.62:33463)
0812 14:25:34.818424 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:34.818425 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:34.818523 (+ 98us) server_negotiation.cc:407] Connection header received
0812 14:25:34.818578 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:34.818579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:34.818589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:34.818639 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:34.819158 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.819642 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.820439 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.820624 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.820678 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:34.821814 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:34.821815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:34.821815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:34.821822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:34.823020 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:34.823331 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:34.823331 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:34.823332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:34.823381 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:34.823547 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:34.823547 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:34.823548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:34.823687 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:34.823766 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:34.823914 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 14:25:34.823979 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:25:34.825655 23822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:25:34.827409 23822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:25:34.834302 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:34.829805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:41509 (local address 127.0.0.1:46158)
0812 14:25:34.829972 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:34.830015 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:34.830080 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:34.830470 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:34.830473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:34.830475 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:34.830549 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:34.830551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.831956 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.831957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:34.833182 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:34.833184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.833977 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.833978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:34.834077 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:34.834094 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 14:25:34.834172 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 14:25:34.834226 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":10,"thread_start_us":50,"threads_started":1}
I20260812 14:25:34.834311 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:34.829901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.5.148.2:41509)
0812 14:25:34.830261 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:34.830266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:34.830277 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:25:34.830322 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:34.830324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:34.830342 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:34.830634 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:34.830720 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.831852 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.833272 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.833884 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.833974 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:34.834041 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:34.834191 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 14:25:34.834230 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":93,"threads_started":1}
I20260812 14:25:34.834434 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:34.829823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:33989 (local address 127.0.0.1:36758)
0812 14:25:34.830450 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:34.830459 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:34.830495 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:34.830932 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:34.830933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:34.830934 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:34.830970 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:34.830971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.832216 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.832216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:34.833202 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:34.833205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.834222 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.834223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:34.834274 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:34.834277 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:25:34.834323 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 14:25:34.834366 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260812 14:25:34.834651 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:34.830112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.5.148.1:33989)
0812 14:25:34.830770 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:34.830774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:34.830785 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:25:34.830823 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:34.830825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:34.830839 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:34.830924 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:34.831033 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.832076 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.833362 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.834087 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.834146 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:34.834193 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:34.834535 (+ 342us) server_negotiation.cc:299] Negotiation successful
0812 14:25:34.834576 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":86,"threads_started":1}
I20260812 14:25:34.834884 24393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:34.830095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.5.148.3:41091)
0812 14:25:34.830309 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:34.830313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:34.830326 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:25:34.830469 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:34.830473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:34.830490 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:34.830567 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:34.830774 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.831695 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.833297 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.834015 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.834090 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:34.834151 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:34.834674 (+ 523us) server_negotiation.cc:299] Negotiation successful
0812 14:25:34.834788 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260812 14:25:34.835032 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:34.829823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:41091 (local address 127.0.0.1:44436)
0812 14:25:34.830275 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:34.830287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:34.830344 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:34.830642 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:34.830643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:34.830645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:34.830687 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:34.830689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.831794 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.831795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:34.833181 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:34.833184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:34.834188 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:34.834189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:34.834254 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:34.834258 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:25:34.834310 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 14:25:34.834352 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":20,"thread_start_us":111,"threads_started":1}
I20260812 14:25:34.835443 24125 tablet_service.cc:1467] Processing CreateTablet for tablet dface8c3221940e1ad2f8e9cc81c754b (DEFAULT_TABLE table=test-table [id=4f2333d024b846bf9ae39c2656f153ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:25:34.835820 24125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dface8c3221940e1ad2f8e9cc81c754b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:34.835805 24282 tablet_service.cc:1467] Processing CreateTablet for tablet dface8c3221940e1ad2f8e9cc81c754b (DEFAULT_TABLE table=test-table [id=4f2333d024b846bf9ae39c2656f153ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:25:34.836191 24282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dface8c3221940e1ad2f8e9cc81c754b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:34.836563 23961 tablet_service.cc:1467] Processing CreateTablet for tablet dface8c3221940e1ad2f8e9cc81c754b (DEFAULT_TABLE table=test-table [id=4f2333d024b846bf9ae39c2656f153ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:25:34.836887 23961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dface8c3221940e1ad2f8e9cc81c754b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:34.838778 24395 tablet_bootstrap.cc:492] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45: Bootstrap starting.
I20260812 14:25:34.839387 24395 tablet_bootstrap.cc:654] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:34.839865 24396 tablet_bootstrap.cc:492] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5: Bootstrap starting.
I20260812 14:25:34.840046 24395 log.cc:824] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:34.840600 24396 tablet_bootstrap.cc:654] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:34.840612 24398 tablet_bootstrap.cc:492] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: Bootstrap starting.
I20260812 14:25:34.840929 24395 tablet_bootstrap.cc:492] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45: No bootstrap required, opened a new log
I20260812 14:25:34.841014 24395 ts_tablet_manager.cc:1397] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:25:34.841241 24396 log.cc:824] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:34.841296 24398 tablet_bootstrap.cc:654] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:34.841835 24398 log.cc:824] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:34.842095 24396 tablet_bootstrap.cc:492] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5: No bootstrap required, opened a new log
I20260812 14:25:34.842165 24396 ts_tablet_manager.cc:1397] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:25:34.842626 24398 tablet_bootstrap.cc:492] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: No bootstrap required, opened a new log
I20260812 14:25:34.842696 24398 ts_tablet_manager.cc:1397] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:25:34.842983 24395 raft_consensus.cc:348] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } } peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } }
I20260812 14:25:34.843272 24395 raft_consensus.cc:374] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:34.843362 24395 raft_consensus.cc:729] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20d1274fcae4bd5a72e189cc1b75f45, State: Initialized, Role: FOLLOWER
I20260812 14:25:34.843497 24395 consensus_queue.cc:260] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } } peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } }
I20260812 14:25:34.843789 24395 ts_tablet_manager.cc:1428] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:25:34.843935 24204 heartbeater.cc:503] Master 127.5.148.62:33463 was elected leader, sending a full tablet report...
I20260812 14:25:34.844308 24396 raft_consensus.cc:348] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } } peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } }
I20260812 14:25:34.844487 24396 raft_consensus.cc:374] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:34.844561 24396 raft_consensus.cc:729] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a09dfb1718a644909deaaae117d972d5, State: Initialized, Role: FOLLOWER
I20260812 14:25:34.844681 24396 consensus_queue.cc:260] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } } peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } }
I20260812 14:25:34.844725 24398 raft_consensus.cc:348] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } } peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } }
I20260812 14:25:34.844894 24398 raft_consensus.cc:374] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:34.844935 24398 raft_consensus.cc:729] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4fbea27ce14ff58406d44e57b14293, State: Initialized, Role: FOLLOWER
I20260812 14:25:34.845012 24396 ts_tablet_manager.cc:1428] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 14:25:34.845062 24398 consensus_queue.cc:260] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } } peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } }
I20260812 14:25:34.845153 24370 heartbeater.cc:503] Master 127.5.148.62:33463 was elected leader, sending a full tablet report...
I20260812 14:25:34.845403 24398 ts_tablet_manager.cc:1428] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:25:34.845407 24044 heartbeater.cc:503] Master 127.5.148.62:33463 was elected leader, sending a full tablet report...
W20260812 14:25:34.894723 24206 tablet.cc:2369] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:25:34.981180 24045 tablet.cc:2369] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:25:35.051677 24373 tablet.cc:2369] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:25:35.144536 24403 raft_consensus.cc:482] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:25:35.144723 24403 raft_consensus.cc:504] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } } peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } }
I20260812 14:25:35.145166 24403 leader_election.cc:288] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa4fbea27ce14ff58406d44e57b14293 (127.5.148.1:33989), a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091)
I20260812 14:25:35.148094 24393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:35.145566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:56067 (local address 127.5.148.3:41091)
0812 14:25:35.145628 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:35.145630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:35.145646 (+ 16us) server_negotiation.cc:407] Connection header received
0812 14:25:35.145672 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:35.145672 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:35.145682 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:35.145750 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:35.145925 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:35.146396 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:35.147328 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:35.147715 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:35.147802 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:35.147814 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:35.147980 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 14:25:35.148019 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 14:25:35.148147 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:35.145540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:56961 (local address 127.5.148.1:33989)
0812 14:25:35.145607 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:35.145609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:35.145620 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:25:35.145692 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:35.145693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:35.145707 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:35.145781 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:35.145969 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:35.146444 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:35.147406 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:35.147764 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:35.147849 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:35.147859 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:35.148014 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 14:25:35.148053 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 14:25:35.148155 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:35.145379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:41091 (local address 127.5.148.2:56067)
0812 14:25:35.145475 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:35.145489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:35.145541 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:35.145757 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:35.145758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:35.145760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:35.145837 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:35.145839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:35.146491 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:35.146492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:35.147224 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:35.147226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:35.147804 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:35.147805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:35.147883 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:35.147886 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:25:35.147968 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 14:25:35.148011 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 14:25:35.148165 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:35.145387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:33989 (local address 127.5.148.2:56961)
0812 14:25:35.145559 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:35.145570 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:35.145628 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:35.145800 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:35.145801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:35.145802 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:35.145892 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:35.145893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:35.146533 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:35.146534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:35.147301 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:35.147304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:35.147858 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:35.147859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:35.147934 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:35.147937 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:25:35.148006 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 14:25:35.148046 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":21,"thread_start_us":70,"threads_started":1}
I20260812 14:25:35.148638 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dface8c3221940e1ad2f8e9cc81c754b" candidate_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a09dfb1718a644909deaaae117d972d5" is_pre_election: true
I20260812 14:25:35.148751 23985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dface8c3221940e1ad2f8e9cc81c754b" candidate_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4fbea27ce14ff58406d44e57b14293" is_pre_election: true
I20260812 14:25:35.148833 24312 raft_consensus.cc:2454] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20d1274fcae4bd5a72e189cc1b75f45 in term 0.
I20260812 14:25:35.148929 23985 raft_consensus.cc:2454] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20d1274fcae4bd5a72e189cc1b75f45 in term 0.
I20260812 14:25:35.149116 24069 leader_election.cc:302] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a09dfb1718a644909deaaae117d972d5, d20d1274fcae4bd5a72e189cc1b75f45; no voters:
I20260812 14:25:35.149276 24403 raft_consensus.cc:2781] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:25:35.149363 24403 raft_consensus.cc:482] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:25:35.149382 24403 raft_consensus.cc:3037] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:35.150141 24403 raft_consensus.cc:504] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } } peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } }
I20260812 14:25:35.150295 24403 leader_election.cc:288] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [CANDIDATE]: Term 1 election: Requested vote from peers fa4fbea27ce14ff58406d44e57b14293 (127.5.148.1:33989), a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091)
I20260812 14:25:35.150454 23985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dface8c3221940e1ad2f8e9cc81c754b" candidate_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4fbea27ce14ff58406d44e57b14293"
I20260812 14:25:35.150499 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dface8c3221940e1ad2f8e9cc81c754b" candidate_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a09dfb1718a644909deaaae117d972d5"
I20260812 14:25:35.150553 23985 raft_consensus.cc:3037] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:35.150602 24312 raft_consensus.cc:3037] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:35.151285 23985 raft_consensus.cc:2454] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20d1274fcae4bd5a72e189cc1b75f45 in term 1.
I20260812 14:25:35.151443 24312 raft_consensus.cc:2454] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20d1274fcae4bd5a72e189cc1b75f45 in term 1.
I20260812 14:25:35.151561 24072 leader_election.cc:302] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d20d1274fcae4bd5a72e189cc1b75f45, fa4fbea27ce14ff58406d44e57b14293; no voters:
I20260812 14:25:35.151692 24403 raft_consensus.cc:2781] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:25:35.151914 24403 raft_consensus.cc:686] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [term 1 LEADER]: Becoming Leader. State: Replica: d20d1274fcae4bd5a72e189cc1b75f45, State: Running, Role: LEADER
I20260812 14:25:35.152050 24403 consensus_queue.cc:237] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } } peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } }
I20260812 14:25:35.153004 23822 catalog_manager.cc:5495] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2). New cstate: current_term: 1 leader_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } health_report { overall_health: UNKNOWN } } }
I20260812 14:25:35.189282 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:35.186969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46164 (local address 127.5.148.2:41509)
0812 14:25:35.187032 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:35.187033 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:35.187042 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:25:35.187083 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:35.187084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:35.187107 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:35.187176 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:25:35.187368 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:35.187854 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:35.188620 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:35.188810 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:35.188870 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:35.189005 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:25:35.189110 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:25:35.189173 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:35.189203 (+ 30us) server_negotiation.cc:299] Negotiation successful
0812 14:25:35.189222 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:25:35.191250 23985 raft_consensus.cc:1261] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Refusing update from remote peer d20d1274fcae4bd5a72e189cc1b75f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:25:35.191250 24312 raft_consensus.cc:1261] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5 [term 1 FOLLOWER]: Refusing update from remote peer d20d1274fcae4bd5a72e189cc1b75f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:25:35.191684 24408 consensus_queue.cc:1035] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:25:35.191843 24403 consensus_queue.cc:1035] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:25:35.194320 24411 mvcc.cc:205] Tried to move back new op lower bound from 7317687235340095488 to 7317687235183730688. Current Snapshot: MvccSnapshot[applied={T|T < 7317687235340095488}]
I20260812 14:25:35.194597 24414 mvcc.cc:205] Tried to move back new op lower bound from 7317687235340095488 to 7317687235183730688. Current Snapshot: MvccSnapshot[applied={T|T < 7317687235340095488}]
I20260812 14:25:35.197183 23822 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:25:35.197286 23822 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:25:35.198074 23822 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Sending DeleteTablet for 3 replicas of tablet dface8c3221940e1ad2f8e9cc81c754b
I20260812 14:25:35.198446 24125 tablet_service.cc:1514] Processing DeleteTablet for tablet dface8c3221940e1ad2f8e9cc81c754b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:35 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:46158
I20260812 14:25:35.198463 23961 tablet_service.cc:1514] Processing DeleteTablet for tablet dface8c3221940e1ad2f8e9cc81c754b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:35 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:36758
I20260812 14:25:35.198493 24282 tablet_service.cc:1514] Processing DeleteTablet for tablet dface8c3221940e1ad2f8e9cc81c754b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:35 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:44436
I20260812 14:25:35.198693 24423 tablet_replica.cc:330] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: stopping tablet replica
I20260812 14:25:35.198760 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 23892
I20260812 14:25:35.198824 24424 tablet_replica.cc:330] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5: stopping tablet replica
I20260812 14:25:35.198959 24424 raft_consensus.cc:2229] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:25:35.199048 24424 raft_consensus.cc:2258] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:25:35.199187 24423 raft_consensus.cc:2229] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:25:35.199278 24423 raft_consensus.cc:2258] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:25:35.203706 24424 ts_tablet_manager.cc:1905] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:25:35.206103 24424 ts_tablet_manager.cc:1918] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:25:35.206489 24424 log.cc:1196] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/dface8c3221940e1ad2f8e9cc81c754b
I20260812 14:25:35.206502 24422 tablet_replica.cc:330] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45: stopping tablet replica
I20260812 14:25:35.206636 24422 raft_consensus.cc:2229] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:25:35.206895 24422 raft_consensus.cc:2258] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:25:35.207016 24424 ts_tablet_manager.cc:1939] T dface8c3221940e1ad2f8e9cc81c754b P a09dfb1718a644909deaaae117d972d5: Deleting consensus metadata
I20260812 14:25:35.207634 23803 catalog_manager.cc:4841] TS a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091): tablet dface8c3221940e1ad2f8e9cc81c754b (table test-table [id=4f2333d024b846bf9ae39c2656f153ae]) successfully deleted
I20260812 14:25:35.207724 24422 ts_tablet_manager.cc:1905] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:25:35.209604 24422 ts_tablet_manager.cc:1918] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:25:35.209690 24422 log.cc:1196] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/dface8c3221940e1ad2f8e9cc81c754b
I20260812 14:25:35.209971 24422 ts_tablet_manager.cc:1939] T dface8c3221940e1ad2f8e9cc81c754b P d20d1274fcae4bd5a72e189cc1b75f45: Deleting consensus metadata
I20260812 14:25:35.210518 23806 catalog_manager.cc:4841] TS d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2:41509): tablet dface8c3221940e1ad2f8e9cc81c754b (table test-table [id=4f2333d024b846bf9ae39c2656f153ae]) successfully deleted
W20260812 14:25:35.212175 23805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:33989 (error 108)
I20260812 14:25:35.212428 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 24049
I20260812 14:25:35.212721 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:35.212596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:33989 (local address 127.0.0.1:36762)
0812 14:25:35.212654 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:35.212686 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:33989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 14:25:35.215297 23805 catalog_manager.cc:4584] TS fa4fbea27ce14ff58406d44e57b14293 (127.5.148.1:33989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dface8c3221940e1ad2f8e9cc81c754b: Network error: Client connection negotiation failed: client connection to 127.5.148.1:33989: connect: Connection refused (error 111)
I20260812 14:25:35.220329 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 24210
I20260812 14:25:35.229008 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 23787
I20260812 14:25:55.238914 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:33463
--webserver_interface=127.5.148.62
--webserver_port=39057
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:33463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:25:55.323266 24426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:55.323515 24426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:55.323554 24426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:55.324885 24426 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 14:25:55.324954 24426 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 14:25:55.324976 24426 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 14:25:55.324995 24426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:25:55.325029 24426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:55.325062 24426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:55.325093 24426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:25:55.325119 24426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:25:55.326704 24426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:33463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:33463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=39057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:55.327000 24426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:55.327244 24426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:55.337239 24434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:55.337273 24437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:55.337364 24435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:55.337277 24426 server_base.cc:1034] running on GCE node
I20260812 14:25:55.337822 24426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:55.338186 24426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:55.339418 24426 hybrid_clock.cc:648] HybridClock initialized: now 1786544755339410 us; error 38 us; skew 500 ppm
Aug 12 14:25:55 dist-test-slave-8w9v krb5kdc[23774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544755, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:55.347644 24426 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:25:55.348263 24426 webserver.cc:460] Webserver started at http://127.5.148.62:39057/ using document root <none> and password file <none>
I20260812 14:25:55.348537 24426 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:55.348631 24426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:55.350049 24426 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:25:55.350735 24446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:55.350914 24426 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:25:55.351028 24426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "6aedb4d320564078bbf24fce1b0c6f72"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "533a550a20c40400625f8d10136fddc2"
server_key_iv: "d0c094aa6bbda4b415e2bbddcac3fcfc"
server_key_version: "encryptionkey@0"
I20260812 14:25:55.351451 24426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:55.367479 24426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:55.369993 24426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:55.370173 24426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:55.374570 24426 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:33463
I20260812 14:25:55.374610 24510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:33463 every 8 connection(s)
I20260812 14:25:55.375066 24426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:25:55.375718 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 24426
I20260812 14:25:55.376067 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:33989
--local_ip_for_outbound_sockets=127.5.148.1
--tserver_master_addrs=127.5.148.62:33463
--webserver_port=41985
--webserver_interface=127.5.148.1
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:25:55.377350 24511 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:25:55.378345 24511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Bootstrap starting.
I20260812 14:25:55.387596 24511 log.cc:824] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:55.389747 24511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:25:55.390101 24511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Bootstrap complete.
I20260812 14:25:55.392711 24511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } }
I20260812 14:25:55.392949 24511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aedb4d320564078bbf24fce1b0c6f72, State: Initialized, Role: FOLLOWER
I20260812 14:25:55.393126 24511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } }
I20260812 14:25:55.393190 24511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:25:55.393245 24511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:25:55.393294 24511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:25:55.394346 24511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } }
I20260812 14:25:55.394464 24511 leader_election.cc:302] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aedb4d320564078bbf24fce1b0c6f72; no voters:
I20260812 14:25:55.394677 24511 leader_election.cc:288] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 14:25:55.394717 24516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 14:25:55.394897 24516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 2 LEADER]: Becoming Leader. State: Replica: 6aedb4d320564078bbf24fce1b0c6f72, State: Running, Role: LEADER
I20260812 14:25:55.394982 24511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:25:55.395049 24516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } }
I20260812 14:25:55.395419 24518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6aedb4d320564078bbf24fce1b0c6f72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } } }
I20260812 14:25:55.395474 24516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aedb4d320564078bbf24fce1b0c6f72. Latest consensus state: current_term: 2 leader_uuid: "6aedb4d320564078bbf24fce1b0c6f72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } } }
I20260812 14:25:55.395535 24518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:55.395565 24516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [sys.catalog]: This master's current role is: LEADER
I20260812 14:25:55.397153 24534 catalog_manager.cc:1259] Loaded cluster ID: e81b7c0d420a48f680caa4b01ca878b6
I20260812 14:25:55.397210 24534 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: loading cluster ID for follower catalog manager: success
I20260812 14:25:55.398372 24534 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: acquiring CA information for follower catalog manager: success
I20260812 14:25:55.398799 24534 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:25:55.398882 24524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:25:55.399226 24524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:25:55.399327 24524 catalog_manager.cc:1259] Loaded cluster ID: e81b7c0d420a48f680caa4b01ca878b6
I20260812 14:25:55.399360 24524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:25:55.399508 24524 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:25:55.399600 24524 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Loaded TSK: 0
I20260812 14:25:55.411924 24524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Generated new TSK 1
I20260812 14:25:55.412045 24524 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:25:55.464668 24514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:55.464938 24514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:55.464980 24514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:55.466187 24514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:55.466256 24514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:55.466303 24514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:25:55.467860 24514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:33989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=41985
--tserver_master_addrs=127.5.148.62:33463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:55.468182 24514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:55.468441 24514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:55.474638 24541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:55.474653 24542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:55.474798 24544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:55.474942 24514 server_base.cc:1034] running on GCE node
I20260812 14:25:55.475167 24514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:55.475466 24514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:55.476624 24514 hybrid_clock.cc:648] HybridClock initialized: now 1786544755476605 us; error 37 us; skew 500 ppm
Aug 12 14:25:55 dist-test-slave-8w9v krb5kdc[23774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544755, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:55.479431 24514 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:25:55.480000 24514 webserver.cc:460] Webserver started at http://127.5.148.1:41985/ using document root <none> and password file <none>
I20260812 14:25:55.480298 24514 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:55.480391 24514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:55.482235 24514 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:25:55.483073 24552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:55.483274 24514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:55.483392 24514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "fa4fbea27ce14ff58406d44e57b14293"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "5a25b2e7b8266833dbfe22ecf51c364d"
server_key_iv: "277b5388e7eee5ed1be01d4e572d15d2"
server_key_version: "encryptionkey@0"
I20260812 14:25:55.483865 24514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:55.508535 24514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:55.510855 24514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:55.511040 24514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:55.511322 24514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:55.511863 24561 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 14:25:55.513442 24514 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 14:25:55.513495 24514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 14:25:55.513531 24514 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 14:25:55.514217 24514 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 14:25:55.514266 24514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 14:25:55.514333 24561 tablet_bootstrap.cc:492] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: Bootstrap starting.
I20260812 14:25:55.523242 24514 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:33989
I20260812 14:25:55.523766 24514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:25:55.527185 24561 log.cc:824] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:55.528666 24687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:33989 every 8 connection(s)
I20260812 14:25:55.530218 24561 tablet_bootstrap.cc:492] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:25:55.530583 24561 tablet_bootstrap.cc:492] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: Bootstrap complete.
I20260812 14:25:55.531970 24561 ts_tablet_manager.cc:1397] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.005s
I20260812 14:25:55.532784 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 24514
I20260812 14:25:55.533179 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:41509
--local_ip_for_outbound_sockets=127.5.148.2
--tserver_master_addrs=127.5.148.62:33463
--webserver_port=44557
--webserver_interface=127.5.148.2
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:25:55.534088 24561 raft_consensus.cc:348] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } } peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } }
I20260812 14:25:55.534510 24561 raft_consensus.cc:729] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4fbea27ce14ff58406d44e57b14293, State: Initialized, Role: FOLLOWER
Aug 12 14:25:55 dist-test-slave-8w9v krb5kdc[23774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544755, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:55.534732 24561 consensus_queue.cc:260] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } } peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } }
I20260812 14:25:55.535064 24561 ts_tablet_manager.cc:1428] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20260812 14:25:55.539772 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.524599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:48967)
0812 14:25:55.527730 (+ 3131us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:55.527783 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:55.528407 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:55.529042 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:55.529047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:55.529326 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:55.529522 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.529527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.530823 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.530824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.531629 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.531631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.532374 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.532376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.532525 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.533244 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:55.533256 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:55.535147 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:55.537640 (+ 2493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:55.537648 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:55.537650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:55.537986 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:55.538280 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:55.538281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:55.538281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:55.538417 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:55.538971 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:55.538979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:55.539179 (+ 200us) client_negotiation.cc:769] Sending connection context
0812 14:25:55.539301 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 14:25:55.539503 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3084,"thread_start_us":106,"threads_started":1}
I20260812 14:25:55.540723 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.524748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:48967 (local address 127.5.148.62:33463)
0812 14:25:55.524971 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:55.524981 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:55.527927 (+ 2946us) server_negotiation.cc:407] Connection header received
0812 14:25:55.528665 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:55.528673 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:55.528737 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:55.528898 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:55.529661 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.530705 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.531734 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.532240 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.532390 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.535267 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:55.535271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:55.535272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:55.535286 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:55.537545 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:55.538082 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:55.538086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:55.538086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:55.538208 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:55.538500 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:55.538502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:55.538502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:55.538847 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:55.538938 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:55.540321 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0812 14:25:55.540485 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":115,"threads_started":1}
I20260812 14:25:55.541575 24688 heartbeater.cc:347] Connected to a master server at 127.5.148.62:33463
I20260812 14:25:55.541682 24688 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:55.541893 24688 heartbeater.cc:511] Master 127.5.148.62:33463 requested a full tablet report, sending...
I20260812 14:25:55.542752 24469 ts_manager.cc:194] Registered new tserver with Master: fa4fbea27ce14ff58406d44e57b14293 (127.5.148.1:33989)
I20260812 14:25:55.543999 24469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:48967
I20260812 14:25:55.545202 24688 heartbeater.cc:503] Master 127.5.148.62:33463 was elected leader, sending a full tablet report...
Aug 12 14:25:55 dist-test-slave-8w9v krb5kdc[23774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544755, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for kudu/127.5.148.1@KRBTEST.COM
I20260812 14:25:55.552261 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.543439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:33989 (local address 127.0.0.1:46496)
0812 14:25:55.543704 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:55.543755 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:55.543831 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:55.544216 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:55.544222 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:55.544610 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:55.544672 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.544675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.545693 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.545694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.546375 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.546378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.546910 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.546911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.547003 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.547573 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:55.547577 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:55.549181 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:55.551014 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:55.551021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:55.551023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:55.551294 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:55.551523 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:55.551524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:55.551525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:55.551594 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:55.551855 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:55.551863 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:55.551919 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 14:25:55.552011 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 14:25:55.552158 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":102,"threads_started":1}
I20260812 14:25:55.552268 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.543518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.5.148.1:33989)
0812 14:25:55.543973 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:55.543977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:55.543988 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:25:55.544023 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:55.544025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:55.544041 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:55.544141 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:55.544793 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.545603 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.546487 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.546795 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.546908 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.549316 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:55.549324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:55.549325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:55.549331 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:55.550931 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:55.551396 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:55.551399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:55.551399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:55.551456 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:55.551656 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:55.551656 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:55.551657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:55.551792 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:55.551857 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:55.552035 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 14:25:55.552110 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":79,"threads_started":1}
I20260812 14:25:55.552703 24609 tablet_service.cc:1514] Processing DeleteTablet for tablet dface8c3221940e1ad2f8e9cc81c754b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:35 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:46496
I20260812 14:25:55.553435 24701 tablet_replica.cc:330] stopping tablet replica
I20260812 14:25:55.553589 24701 raft_consensus.cc:2229] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:25:55.553679 24701 raft_consensus.cc:2258] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:25:55.554016 24701 ts_tablet_manager.cc:1905] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:25:55.556404 24701 ts_tablet_manager.cc:1918] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:25:55.556499 24701 log.cc:1196] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal/wals/dface8c3221940e1ad2f8e9cc81c754b
I20260812 14:25:55.556742 24701 ts_tablet_manager.cc:1939] T dface8c3221940e1ad2f8e9cc81c754b P fa4fbea27ce14ff58406d44e57b14293: Deleting consensus metadata
I20260812 14:25:55.557269 24451 catalog_manager.cc:4841] TS fa4fbea27ce14ff58406d44e57b14293 (127.5.148.1:33989): tablet dface8c3221940e1ad2f8e9cc81c754b (table test-table [id=4f2333d024b846bf9ae39c2656f153ae]) successfully deleted
W20260812 14:25:55.625023 24696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:55.625226 24696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:55.625250 24696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:55.626524 24696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:55.626569 24696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:55.626596 24696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:25:55.628093 24696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:41509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=44557
--tserver_master_addrs=127.5.148.62:33463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:55.628391 24696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:55.628638 24696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:55.633883 24708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:55.633842 24707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:55.633838 24711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:55.634162 24696 server_base.cc:1034] running on GCE node
I20260812 14:25:55.634413 24696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:55.634702 24696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:55.635862 24696 hybrid_clock.cc:648] HybridClock initialized: now 1786544755635847 us; error 48 us; skew 500 ppm
Aug 12 14:25:55 dist-test-slave-8w9v krb5kdc[23774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544755, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:55.638576 24696 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:25:55.639048 24696 webserver.cc:460] Webserver started at http://127.5.148.2:44557/ using document root <none> and password file <none>
I20260812 14:25:55.639248 24696 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:55.639309 24696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:55.640832 24696 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:55.641471 24720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:55.641670 24696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:55.641769 24696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "d20d1274fcae4bd5a72e189cc1b75f45"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "b9f60de51d52d45ef2e64b06690cc1da"
server_key_iv: "2b52d0edef6de98b1ef8309c35a5f941"
server_key_version: "encryptionkey@0"
I20260812 14:25:55.642153 24696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:55.651480 24696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:55.653932 24696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:55.654107 24696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:55.654346 24696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:55.654660 24696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:55.654695 24696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:55.654717 24696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:55.654731 24696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:55.660835 24696 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:41509
I20260812 14:25:55.660879 24859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:41509 every 8 connection(s)
I20260812 14:25:55.661230 24696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:25:55 dist-test-slave-8w9v krb5kdc[23774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544755, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:55.670446 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.662089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:39365 (local address 127.5.148.62:33463)
0812 14:25:55.662153 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:55.662158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:55.662360 (+ 202us) server_negotiation.cc:407] Connection header received
0812 14:25:55.662835 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:55.662836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:55.662848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:55.662909 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:55.663824 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.664382 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.665406 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.665610 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.665666 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.667865 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:55.667866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:55.667866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:55.667872 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:55.669164 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:55.669545 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:55.669546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:55.669546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:55.669599 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:55.669814 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:55.669815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:55.669815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:55.669971 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:55.670046 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:55.670298 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 14:25:55.670379 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:25:55.670610 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.662041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:39365)
0812 14:25:55.662203 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:55.662248 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:55.662749 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:55.662970 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:55.662973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:55.663550 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:55.663746 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.663751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.664511 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.664513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.665321 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.665324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.665692 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.665693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.665797 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.666369 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:55.666378 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:55.667781 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:55.669239 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:55.669246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:55.669248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:55.669477 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:55.669659 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:55.669659 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:55.669659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:55.669759 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:55.670042 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:55.670046 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:55.670182 (+ 136us) client_negotiation.cc:769] Sending connection context
0812 14:25:55.670267 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 14:25:55.670387 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":107,"threads_started":1}
I20260812 14:25:55.670919 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 24696
I20260812 14:25:55.671268 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:41091
--local_ip_for_outbound_sockets=127.5.148.3
--tserver_master_addrs=127.5.148.62:33463
--webserver_port=40453
--webserver_interface=127.5.148.3
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:25:55.671271 24860 heartbeater.cc:347] Connected to a master server at 127.5.148.62:33463
I20260812 14:25:55.671370 24860 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:55.671566 24860 heartbeater.cc:511] Master 127.5.148.62:33463 requested a full tablet report, sending...
I20260812 14:25:55.671957 24469 ts_manager.cc:194] Registered new tserver with Master: d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2:41509)
I20260812 14:25:55.672405 24469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:39365
W20260812 14:25:55.759204 24864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:25:55.759420 24864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:25:55.759447 24864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:25:55.760700 24864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:25:55.760743 24864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:25:55.760797 24864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:25:55.762315 24864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:41091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=40453
--tserver_master_addrs=127.5.148.62:33463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:25:55.762657 24864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:25:55.762912 24864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:25:55.768198 24871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:25:55.768188 24873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:55.768275 24864 server_base.cc:1034] running on GCE node
W20260812 14:25:55.768189 24869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:25:55.768699 24864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:25:55.768946 24864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:25:55.770143 24864 hybrid_clock.cc:648] HybridClock initialized: now 1786544755770126 us; error 45 us; skew 500 ppm
Aug 12 14:25:55 dist-test-slave-8w9v krb5kdc[23774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544755, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:25:55.772753 24864 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:25:55.773227 24864 webserver.cc:460] Webserver started at http://127.5.148.3:40453/ using document root <none> and password file <none>
I20260812 14:25:55.773463 24864 fs_manager.cc:359] Metadata directory not provided
I20260812 14:25:55.773546 24864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:25:55.774914 24864 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:25:55.775499 24881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:55.775674 24864 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:55.775781 24864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "a09dfb1718a644909deaaae117d972d5"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "9762d105a8c907c115cd186b4aa2a117"
server_key_iv: "5b5ec19cc5ea48696dbfa8952f95bd25"
server_key_version: "encryptionkey@0"
I20260812 14:25:55.776162 24864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:25:55.787525 24864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:25:55.789762 24864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:25:55.789932 24864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:25:55.790167 24864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:25:55.790489 24864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:25:55.790544 24864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:55.790587 24864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:25:55.790613 24864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:25:55.796909 24864 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:41091
I20260812 14:25:55.797051 25018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:41091 every 8 connection(s)
I20260812 14:25:55.797369 24864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:25:55 dist-test-slave-8w9v krb5kdc[23774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544755, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:25:55.806650 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.798166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:51825 (local address 127.5.148.62:33463)
0812 14:25:55.798232 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:55.798235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:55.798497 (+ 262us) server_negotiation.cc:407] Connection header received
0812 14:25:55.798984 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:55.798985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:55.798996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:55.799051 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:55.800001 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.800583 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.801583 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.801774 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.801827 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.804159 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:55.804160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:55.804160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:55.804166 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:55.805437 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:55.805820 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:55.805821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:55.805821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:55.805877 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:55.806101 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:55.806101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:55.806102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:55.806204 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:55.806255 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:55.806518 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 14:25:55.806584 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:25:55.806826 25021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.798162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:51825)
0812 14:25:55.798358 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:55.798395 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:55.798897 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:55.799108 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:55.799111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:55.799716 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:25:55.799916 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.799923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.800668 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.800669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.801502 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.801504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.801852 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.801853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.801963 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.802606 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:25:55.802615 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:25:55.804045 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:25:55.805513 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:55.805520 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:55.805522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:55.805754 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:55.805942 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:25:55.805942 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:25:55.805943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:25:55.806044 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:25:55.806261 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:25:55.806266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:25:55.806389 (+ 123us) client_negotiation.cc:769] Sending connection context
0812 14:25:55.806489 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 14:25:55.806616 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
I20260812 14:25:55.807400 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 24864
I20260812 14:25:55.807446 25019 heartbeater.cc:347] Connected to a master server at 127.5.148.62:33463
I20260812 14:25:55.807574 25019 heartbeater.cc:464] Registering TS with master...
I20260812 14:25:55.807781 25019 heartbeater.cc:511] Master 127.5.148.62:33463 requested a full tablet report, sending...
I20260812 14:25:55.808261 24469 ts_manager.cc:194] Registered new tserver with Master: a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091)
I20260812 14:25:55.808647 24469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:51825
I20260812 14:25:55.814266 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.807869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34272 (local address 127.5.148.62:33463)
0812 14:25:55.807927 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:55.807929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:55.808159 (+ 230us) server_negotiation.cc:407] Connection header received
0812 14:25:55.808271 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:55.808272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:55.808283 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:55.808363 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:55.809071 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.809704 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.810416 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.810588 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.810638 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.812240 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:55.812241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:55.812241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:55.812247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:55.813367 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:55.813678 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:55.813678 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:55.813679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:55.813727 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:55.813900 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:55.813900 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:55.813900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:55.813992 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:55.814108 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:55.814154 (+ 46us) server_negotiation.cc:299] Negotiation successful
0812 14:25:55.814210 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:25:55.815043 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:25:55.818081 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.815403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.5.148.62:33463)
0812 14:25:55.815479 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:55.815481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:55.815579 (+ 98us) server_negotiation.cc:407] Connection header received
0812 14:25:55.815665 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:55.815666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:55.815708 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:55.815786 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:25:55.815942 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.816496 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.817244 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.817512 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.817589 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.817776 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:25:55.817832 (+ 56us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 14:25:55.818013 (+ 181us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34284: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260812 14:25:55.818136 24694 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34284: token expired
W20260812 14:25:55.818341 24376 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.148.62:33463)
I20260812 14:25:55.824057 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.818652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34294 (local address 127.5.148.62:33463)
0812 14:25:55.818705 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:55.818707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:55.818735 (+ 28us) server_negotiation.cc:407] Connection header received
0812 14:25:55.818782 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:55.818783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:55.818790 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:55.818836 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:25:55.819279 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.819709 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.820432 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.820600 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.820651 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.821721 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:25:55.821723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:25:55.821723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:25:55.821729 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:25:55.823155 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:55.823477 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:55.823478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:55.823478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:55.823528 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:25:55.823692 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:25:55.823693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:25:55.823693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:25:55.823786 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:25:55.823857 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:55.823944 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 14:25:55.824001 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:25:55.825336 24469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:25:55.827143 24469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:25:55.829456 24609 tablet_service.cc:1467] Processing CreateTablet for tablet a4552c68ca1d4a968f704656d4f1d7a9 (DEFAULT_TABLE table=test-table [id=9f37899a6ae54200b4943ad5ba62e142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:25:55.829703 24609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4552c68ca1d4a968f704656d4f1d7a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:55.831106 24561 tablet_bootstrap.cc:492] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293: Bootstrap starting.
I20260812 14:25:55.831475 24561 tablet_bootstrap.cc:654] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:55.832084 24561 tablet_bootstrap.cc:492] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293: No bootstrap required, opened a new log
I20260812 14:25:55.832154 24561 ts_tablet_manager.cc:1397] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260812 14:25:55.832551 24561 raft_consensus.cc:348] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:25:55.832640 24561 raft_consensus.cc:374] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:55.832705 24561 raft_consensus.cc:729] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4fbea27ce14ff58406d44e57b14293, State: Initialized, Role: FOLLOWER
I20260812 14:25:55.832757 24561 consensus_queue.cc:260] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:25:55.832851 24561 ts_tablet_manager.cc:1428] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:25:55.833853 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.829141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:41091 (local address 127.0.0.1:37230)
0812 14:25:55.829614 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:55.829626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:55.829691 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:55.829912 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:55.829913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:55.829916 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:55.829962 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.829964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.830960 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.830962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.832472 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.832476 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.833593 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.833594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.833658 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.833680 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 14:25:55.833736 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 14:25:55.833777 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":343,"thread_start_us":62,"threads_started":1}
I20260812 14:25:55.833876 25025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.829286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37230 (local address 127.5.148.3:41091)
0812 14:25:55.829504 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:55.829509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:55.829690 (+ 181us) server_negotiation.cc:407] Connection header received
0812 14:25:55.829783 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:55.829786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:55.829805 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:55.829889 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:55.830084 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.830871 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.832578 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.833492 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.833590 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.833655 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:55.833750 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 14:25:55.833788 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":112,"threads_started":1}
I20260812 14:25:55.834503 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.829142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:41509 (local address 127.0.0.1:45730)
0812 14:25:55.829179 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:55.829191 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:55.829231 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:55.831501 (+ 2270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:55.831502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:55.831504 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:55.831551 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.831553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.832652 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.832653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.833454 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.833456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.834237 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.834238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.834331 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.834343 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 14:25:55.834400 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 14:25:55.834459 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260812 14:25:55.834599 25026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.829306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.5.148.2:41509)
0812 14:25:55.831335 (+ 2029us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:55.831339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:55.831349 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:25:55.831391 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:55.831393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:55.831405 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:55.831504 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:55.831600 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.832570 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.833562 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.834163 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.834234 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.834290 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:55.834470 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 14:25:55.834514 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2000,"thread_start_us":119,"threads_started":1}
I20260812 14:25:55.834630 24937 tablet_service.cc:1467] Processing CreateTablet for tablet a4552c68ca1d4a968f704656d4f1d7a9 (DEFAULT_TABLE table=test-table [id=9f37899a6ae54200b4943ad5ba62e142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:25:55.835011 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4552c68ca1d4a968f704656d4f1d7a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:55.835237 24776 tablet_service.cc:1467] Processing CreateTablet for tablet a4552c68ca1d4a968f704656d4f1d7a9 (DEFAULT_TABLE table=test-table [id=9f37899a6ae54200b4943ad5ba62e142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:25:55.835543 24776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4552c68ca1d4a968f704656d4f1d7a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:25:55.837858 25030 tablet_bootstrap.cc:492] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5: Bootstrap starting.
I20260812 14:25:55.838502 25031 tablet_bootstrap.cc:492] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45: Bootstrap starting.
I20260812 14:25:55.838619 25030 tablet_bootstrap.cc:654] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:55.839202 25031 tablet_bootstrap.cc:654] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45: Neither blocks nor log segments found. Creating new log.
I20260812 14:25:55.839246 25030 log.cc:824] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:55.839763 25031 log.cc:824] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45: Log is configured to *not* fsync() on all Append() calls
I20260812 14:25:55.840122 25030 tablet_bootstrap.cc:492] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5: No bootstrap required, opened a new log
I20260812 14:25:55.840225 25030 ts_tablet_manager.cc:1397] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:25:55.840565 25031 tablet_bootstrap.cc:492] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45: No bootstrap required, opened a new log
I20260812 14:25:55.840648 25031 ts_tablet_manager.cc:1397] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:25:55.841581 25030 raft_consensus.cc:348] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:25:55.841722 25030 raft_consensus.cc:374] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:55.841745 25030 raft_consensus.cc:729] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a09dfb1718a644909deaaae117d972d5, State: Initialized, Role: FOLLOWER
I20260812 14:25:55.841869 25030 consensus_queue.cc:260] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:25:55.842162 25030 ts_tablet_manager.cc:1428] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:25:55.842192 25019 heartbeater.cc:503] Master 127.5.148.62:33463 was elected leader, sending a full tablet report...
I20260812 14:25:55.842198 25031 raft_consensus.cc:348] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:25:55.842374 25031 raft_consensus.cc:374] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:25:55.842420 25031 raft_consensus.cc:729] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20d1274fcae4bd5a72e189cc1b75f45, State: Initialized, Role: FOLLOWER
I20260812 14:25:55.842536 25031 consensus_queue.cc:260] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:25:55.842833 25031 ts_tablet_manager.cc:1428] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:25:55.842851 24860 heartbeater.cc:503] Master 127.5.148.62:33463 was elected leader, sending a full tablet report...
W20260812 14:25:55.911856 24861 tablet.cc:2369] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:25:55.937103 25035 raft_consensus.cc:482] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:25:55.937242 25035 raft_consensus.cc:504] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:25:55.937644 25035 leader_election.cc:288] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091), fa4fbea27ce14ff58406d44e57b14293 (127.5.148.1:33989)
I20260812 14:25:55.940725 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.937829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:41091 (local address 127.5.148.2:44609)
0812 14:25:55.937871 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:55.937886 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:55.937939 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:55.938152 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:55.938154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:55.938156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:55.938238 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.938240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.938951 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.938952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.939760 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.939762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.940443 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.940444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.940532 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.940537 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:25:55.940593 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 14:25:55.940635 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260812 14:25:55.940755 25025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.937909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:44609 (local address 127.5.148.3:41091)
0812 14:25:55.937991 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:55.937992 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:55.938002 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:25:55.938038 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:55.938039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:55.938053 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:55.938115 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:55.938324 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.938860 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.939861 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.940361 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.940421 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.940431 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:55.940614 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 14:25:55.940658 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 14:25:55.940887 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.937920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:44087 (local address 127.5.148.1:33989)
0812 14:25:55.937972 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:55.937975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:55.938129 (+ 154us) server_negotiation.cc:407] Connection header received
0812 14:25:55.938160 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:55.938161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:55.938191 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:55.938279 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:25:55.938485 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.939027 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.939979 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.940492 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.940567 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.940654 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:55.940752 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 14:25:55.940795 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:25:55.941092 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.937835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:33989 (local address 127.5.148.2:44087)
0812 14:25:55.938045 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:55.938055 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:25:55.938105 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:25:55.938314 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:25:55.938315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:25:55.938317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:25:55.938396 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.938397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.939116 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.939117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.939886 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:25:55.939888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.940597 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.940598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:25:55.940673 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.940678 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:25:55.940739 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:25:55.940998 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":14,"thread_start_us":121,"threads_started":1}
I20260812 14:25:55.941128 24960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4552c68ca1d4a968f704656d4f1d7a9" candidate_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a09dfb1718a644909deaaae117d972d5" is_pre_election: true
I20260812 14:25:55.941288 24960 raft_consensus.cc:2454] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20d1274fcae4bd5a72e189cc1b75f45 in term 0.
I20260812 14:25:55.941432 24624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4552c68ca1d4a968f704656d4f1d7a9" candidate_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4fbea27ce14ff58406d44e57b14293" is_pre_election: true
I20260812 14:25:55.941606 24624 raft_consensus.cc:2454] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20d1274fcae4bd5a72e189cc1b75f45 in term 0.
I20260812 14:25:55.941597 24721 leader_election.cc:302] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a09dfb1718a644909deaaae117d972d5, d20d1274fcae4bd5a72e189cc1b75f45; no voters:
I20260812 14:25:55.941776 25035 raft_consensus.cc:2781] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:25:55.941859 25035 raft_consensus.cc:482] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:25:55.941910 25035 raft_consensus.cc:3037] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:55.942646 25035 raft_consensus.cc:504] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:25:55.942798 25035 leader_election.cc:288] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [CANDIDATE]: Term 1 election: Requested vote from peers a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091), fa4fbea27ce14ff58406d44e57b14293 (127.5.148.1:33989)
I20260812 14:25:55.942962 24624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4552c68ca1d4a968f704656d4f1d7a9" candidate_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4fbea27ce14ff58406d44e57b14293"
I20260812 14:25:55.942981 24960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4552c68ca1d4a968f704656d4f1d7a9" candidate_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a09dfb1718a644909deaaae117d972d5"
I20260812 14:25:55.943042 24624 raft_consensus.cc:3037] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:55.943069 24960 raft_consensus.cc:3037] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:25:55.943758 24624 raft_consensus.cc:2454] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20d1274fcae4bd5a72e189cc1b75f45 in term 1.
I20260812 14:25:55.943917 24960 raft_consensus.cc:2454] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20d1274fcae4bd5a72e189cc1b75f45 in term 1.
I20260812 14:25:55.943948 24723 leader_election.cc:302] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d20d1274fcae4bd5a72e189cc1b75f45, fa4fbea27ce14ff58406d44e57b14293; no voters:
I20260812 14:25:55.944056 25035 raft_consensus.cc:2781] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:25:55.944255 25035 raft_consensus.cc:686] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [term 1 LEADER]: Becoming Leader. State: Replica: d20d1274fcae4bd5a72e189cc1b75f45, State: Running, Role: LEADER
I20260812 14:25:55.944371 25035 consensus_queue.cc:237] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:25:55.945322 24469 catalog_manager.cc:5495] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2). New cstate: current_term: 1 leader_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } health_report { overall_health: UNKNOWN } } }
I20260812 14:25:55.950038 25026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.947923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45740 (local address 127.5.148.2:41509)
0812 14:25:55.947978 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:25:55.947980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:25:55.947989 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:25:55.948018 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:25:55.948018 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:25:55.948037 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:25:55.948103 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:25:55.948253 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.948699 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.949388 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:25:55.949570 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:25:55.949618 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:25:55.949746 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:25:55.949854 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:25:55.949916 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:25:55.949950 (+ 34us) server_negotiation.cc:299] Negotiation successful
0812 14:25:55.949967 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:25:55.951773 24624 raft_consensus.cc:1261] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Refusing update from remote peer d20d1274fcae4bd5a72e189cc1b75f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:25:55.951822 24960 raft_consensus.cc:1261] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5 [term 1 FOLLOWER]: Refusing update from remote peer d20d1274fcae4bd5a72e189cc1b75f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:25:55.952203 25037 consensus_queue.cc:1035] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:25:55.952344 25035 consensus_queue.cc:1035] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:25:55.954402 25039 mvcc.cc:205] Tried to move back new op lower bound from 7317687320375631872 to 7317687320348839936. Current Snapshot: MvccSnapshot[applied={T|T < 7317687320375631872}]
I20260812 14:25:55.954909 25042 mvcc.cc:205] Tried to move back new op lower bound from 7317687320375631872 to 7317687320348839936. Current Snapshot: MvccSnapshot[applied={T|T < 7317687320375631872}]
I20260812 14:25:55.955343 25041 mvcc.cc:205] Tried to move back new op lower bound from 7317687320375631872 to 7317687320348839936. Current Snapshot: MvccSnapshot[applied={T|T < 7317687320375631872}]
I20260812 14:25:55.957042 24469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:25:55.957135 24469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:25:55.957896 24469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Sending DeleteTablet for 3 replicas of tablet a4552c68ca1d4a968f704656d4f1d7a9
I20260812 14:25:55.958230 24609 tablet_service.cc:1514] Processing DeleteTablet for tablet a4552c68ca1d4a968f704656d4f1d7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:55 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:46496
I20260812 14:25:55.958227 24776 tablet_service.cc:1514] Processing DeleteTablet for tablet a4552c68ca1d4a968f704656d4f1d7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:55 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:45730
I20260812 14:25:55.958230 24937 tablet_service.cc:1514] Processing DeleteTablet for tablet a4552c68ca1d4a968f704656d4f1d7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:25:55 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:37230
I20260812 14:25:55.958428 24701 tablet_replica.cc:330] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293: stopping tablet replica
I20260812 14:25:55.958489 24701 raft_consensus.cc:2229] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:25:55.958554 24701 raft_consensus.cc:2258] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:25:55.958609 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 24514
I20260812 14:25:55.970337 25053 tablet_replica.cc:330] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45: stopping tablet replica
I20260812 14:25:55.970522 25053 raft_consensus.cc:2229] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:25:55.970705 25053 raft_consensus.cc:2258] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:25:55.970932 25052 tablet_replica.cc:330] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5: stopping tablet replica
I20260812 14:25:55.971065 25052 raft_consensus.cc:2229] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:25:55.971185 25052 raft_consensus.cc:2258] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:25:55.971468 25053 ts_tablet_manager.cc:1905] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:25:55.972021 25052 ts_tablet_manager.cc:1905] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:25:55.973727 25053 ts_tablet_manager.cc:1918] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:25:55.973820 25053 log.cc:1196] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/a4552c68ca1d4a968f704656d4f1d7a9
I20260812 14:25:55.973775 25052 ts_tablet_manager.cc:1918] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:25:55.973908 25052 log.cc:1196] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/a4552c68ca1d4a968f704656d4f1d7a9
I20260812 14:25:55.974131 25053 ts_tablet_manager.cc:1939] T a4552c68ca1d4a968f704656d4f1d7a9 P d20d1274fcae4bd5a72e189cc1b75f45: Deleting consensus metadata
I20260812 14:25:55.974265 25052 ts_tablet_manager.cc:1939] T a4552c68ca1d4a968f704656d4f1d7a9 P a09dfb1718a644909deaaae117d972d5: Deleting consensus metadata
I20260812 14:25:55.974643 24452 catalog_manager.cc:4841] TS d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2:41509): tablet a4552c68ca1d4a968f704656d4f1d7a9 (table test-table [id=9f37899a6ae54200b4943ad5ba62e142]) successfully deleted
I20260812 14:25:55.974737 24447 catalog_manager.cc:4841] TS a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091): tablet a4552c68ca1d4a968f704656d4f1d7a9 (table test-table [id=9f37899a6ae54200b4943ad5ba62e142]) successfully deleted
W20260812 14:25:55.975880 24451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:33989 (error 108)
I20260812 14:25:55.976404 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 24696
I20260812 14:25:55.976442 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:25:55.976299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:33989 (local address 127.0.0.1:46506)
0812 14:25:55.976366 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0812 14:25:55.976415 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:33989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260812 14:25:55.982376 24451 catalog_manager.cc:4584] TS fa4fbea27ce14ff58406d44e57b14293 (127.5.148.1:33989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4552c68ca1d4a968f704656d4f1d7a9: Network error: Client connection negotiation failed: client connection to 127.5.148.1:33989: connect: Connection refused (error 111)
I20260812 14:25:55.987712 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 24864
I20260812 14:25:55.994134 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 24426
I20260812 14:26:16.003902 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:33463
--webserver_interface=127.5.148.62
--webserver_port=39057
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:33463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:16.091264 25055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:16.091547 25055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:16.091588 25055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:16.092831 25055 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 14:26:16.092907 25055 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 14:26:16.092931 25055 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 14:26:16.092952 25055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:16.092981 25055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:16.093015 25055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:16.093040 25055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:16.093065 25055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:16.094645 25055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:33463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:33463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=39057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:16.094949 25055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:16.095188 25055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:16.100723 25061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:16.100734 25063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:16.100736 25065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:16.100797 25055 server_base.cc:1034] running on GCE node
I20260812 14:26:16.101188 25055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:16.101588 25055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:16.102777 25055 hybrid_clock.cc:648] HybridClock initialized: now 1786544776102755 us; error 42 us; skew 500 ppm
Aug 12 14:26:16 dist-test-slave-8w9v krb5kdc[23774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544776, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:16.110612 25055 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:26:16.111143 25055 webserver.cc:460] Webserver started at http://127.5.148.62:39057/ using document root <none> and password file <none>
I20260812 14:26:16.111410 25055 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:16.111493 25055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:16.113035 25055 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:16.113750 25074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:16.113956 25055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:16.114073 25055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "6aedb4d320564078bbf24fce1b0c6f72"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "533a550a20c40400625f8d10136fddc2"
server_key_iv: "d0c094aa6bbda4b415e2bbddcac3fcfc"
server_key_version: "encryptionkey@0"
I20260812 14:26:16.114447 25055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:16.139470 25055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:16.141717 25055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:16.141883 25055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:16.146217 25055 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:33463
I20260812 14:26:16.146308 25143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:33463 every 8 connection(s)
I20260812 14:26:16.146689 25055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:16.148301 25145 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:26:16.148955 25145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Bootstrap starting.
I20260812 14:26:16.151470 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 25055
I20260812 14:26:16.151858 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:33989
--local_ip_for_outbound_sockets=127.5.148.1
--tserver_master_addrs=127.5.148.62:33463
--webserver_port=41985
--webserver_interface=127.5.148.1
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:26:16.158793 25145 log.cc:824] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:16.161499 25145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:26:16.161861 25145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Bootstrap complete.
I20260812 14:26:16.164340 25145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } }
I20260812 14:26:16.164608 25145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aedb4d320564078bbf24fce1b0c6f72, State: Initialized, Role: FOLLOWER
I20260812 14:26:16.164835 25145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } }
I20260812 14:26:16.164945 25145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:16.165004 25145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:16.165057 25145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 2 FOLLOWER]: Advancing to term 3
I20260812 14:26:16.166107 25145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } }
I20260812 14:26:16.166241 25145 leader_election.cc:302] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aedb4d320564078bbf24fce1b0c6f72; no voters:
I20260812 14:26:16.166477 25145 leader_election.cc:288] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 14:26:16.166519 25149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 14:26:16.166656 25149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [term 3 LEADER]: Becoming Leader. State: Replica: 6aedb4d320564078bbf24fce1b0c6f72, State: Running, Role: LEADER
I20260812 14:26:16.166791 25145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:16.166917 25149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } }
I20260812 14:26:16.167457 25152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6aedb4d320564078bbf24fce1b0c6f72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } } }
I20260812 14:26:16.167465 25153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aedb4d320564078bbf24fce1b0c6f72. Latest consensus state: current_term: 3 leader_uuid: "6aedb4d320564078bbf24fce1b0c6f72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aedb4d320564078bbf24fce1b0c6f72" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33463 } } }
I20260812 14:26:16.167573 25152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:16.167593 25153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:16.168040 25161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:16.168900 25161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:16.169023 25161 catalog_manager.cc:1259] Loaded cluster ID: e81b7c0d420a48f680caa4b01ca878b6
I20260812 14:26:16.169065 25161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:16.169888 25161 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:16.170080 25161 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Loaded TSK: 0
I20260812 14:26:16.170122 25161 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6aedb4d320564078bbf24fce1b0c6f72: Loaded TSK: 1
F20260812 14:26:16.170348 25161 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc8a34989ad google::LogMessage::Fail() at ??:0
@ 0x7fc8a349b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc8a34984de google::LogMessage::Flush() at ??:0
@ 0x7fc8a3499a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc8a5c05104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc8a5c02d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc8ab3f21cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc8ab3f248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc8ab3f2594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc8ab3b8c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc8ab3c2ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc8a53660d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc8a535b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc8a61bf6db start_thread at ??:0
@ 0x7fc8a1f9661f clone at ??:0
W20260812 14:26:16.258085 25147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:16.258294 25147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:16.258322 25147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:16.259510 25147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:16.259555 25147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:16.259582 25147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:26:16.261170 25147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:33989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=41985
--tserver_master_addrs=127.5.148.62:33463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:16.261426 25147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:16.261682 25147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:16.267917 25196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:16.267894 25198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:16.268111 25147 server_base.cc:1034] running on GCE node
W20260812 14:26:16.268260 25195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:16.268498 25147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:16.268780 25147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:16.269943 25147 hybrid_clock.cc:648] HybridClock initialized: now 1786544776269919 us; error 41 us; skew 500 ppm
Aug 12 14:26:16 dist-test-slave-8w9v krb5kdc[23774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544776, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:16.272694 25147 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:26:16.273245 25147 webserver.cc:460] Webserver started at http://127.5.148.1:41985/ using document root <none> and password file <none>
I20260812 14:26:16.273494 25147 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:16.273541 25147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:16.274960 25147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 14:26:16.275696 25207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:16.275913 25147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:16.276043 25147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "fa4fbea27ce14ff58406d44e57b14293"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "5a25b2e7b8266833dbfe22ecf51c364d"
server_key_iv: "277b5388e7eee5ed1be01d4e572d15d2"
server_key_version: "encryptionkey@0"
I20260812 14:26:16.276572 25147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:16.331712 25147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:16.334321 25147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:16.334510 25147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:16.334772 25147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:16.335261 25216 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 14:26:16.337392 25147 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 14:26:16.337459 25147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 14:26:16.337523 25147 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 14:26:16.338199 25147 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 14:26:16.338258 25147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 14:26:16.338346 25216 tablet_bootstrap.cc:492] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293: Bootstrap starting.
I20260812 14:26:16.346179 25147 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:33989
I20260812 14:26:16.346335 25347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:33989 every 8 connection(s)
I20260812 14:26:16.346647 25147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:26:16.349738 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 25147
I20260812 14:26:16.350054 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:41509
--local_ip_for_outbound_sockets=127.5.148.2
--tserver_master_addrs=127.5.148.62:33463
--webserver_port=44557
--webserver_interface=127.5.148.2
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:26:16.350379 25216 log.cc:824] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:16.353340 25216 tablet_bootstrap.cc:492] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:26:16.353698 25216 tablet_bootstrap.cc:492] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293: Bootstrap complete.
I20260812 14:26:16.355116 25216 ts_tablet_manager.cc:1397] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20260812 14:26:16.356711 25216 raft_consensus.cc:348] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:26:16.357066 25216 raft_consensus.cc:729] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4fbea27ce14ff58406d44e57b14293, State: Initialized, Role: FOLLOWER
I20260812 14:26:16.357226 25216 consensus_queue.cc:260] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:26:16.357555 25216 ts_tablet_manager.cc:1428] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260812 14:26:16.396657 25351 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:26:16.347369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:60923)
0812 14:26:16.347554 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:16.347601 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:16.348111 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:16.396440 (+ 48329us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
W20260812 14:26:16.396956 25348 heartbeater.cc:644] Failed to heartbeat to 127.5.148.62:33463 (0 consecutive failures): Network error: Failed to ping master at 127.5.148.62:33463: Client connection negotiation failed: client connection to 127.5.148.62:33463: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 14:26:16.397329 25351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:16.397202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:56031)
0812 14:26:16.397255 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:16.397304 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 14:26:16.397683 25351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:16.397593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:36615)
0812 14:26:16.397639 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:16.397666 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 14:26:16.397807 25348 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:26:16.440922 25353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:16.441149 25353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:16.441174 25353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:16.442401 25353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:16.442446 25353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:16.442476 25353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:26:16.443974 25353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:41509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=44557
--tserver_master_addrs=127.5.148.62:33463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:16.444248 25353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:16.444507 25353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:16.449674 25364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:16.449673 25360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:16.449729 25362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:16.449829 25353 server_base.cc:1034] running on GCE node
I20260812 14:26:16.450126 25353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:16.450369 25353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:16.451534 25353 hybrid_clock.cc:648] HybridClock initialized: now 1786544776451526 us; error 36 us; skew 500 ppm
Aug 12 14:26:16 dist-test-slave-8w9v krb5kdc[23774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544776, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:16.454259 25353 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:26:16.454751 25353 webserver.cc:460] Webserver started at http://127.5.148.2:44557/ using document root <none> and password file <none>
I20260812 14:26:16.454990 25353 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:16.455062 25353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:16.456600 25353 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:16.457319 25374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:16.457516 25353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:16.457631 25353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "d20d1274fcae4bd5a72e189cc1b75f45"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "b9f60de51d52d45ef2e64b06690cc1da"
server_key_iv: "2b52d0edef6de98b1ef8309c35a5f941"
server_key_version: "encryptionkey@0"
I20260812 14:26:16.458033 25353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:16.490237 25353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:16.492911 25353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:16.493090 25353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:16.493371 25353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:16.493707 25353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:16.493762 25353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:16.493799 25353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:16.493844 25353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:16.500317 25353 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:41509
I20260812 14:26:16.500367 25514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:41509 every 8 connection(s)
I20260812 14:26:16.500727 25353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:26:16.501859 25518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:16.501415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:40457)
0812 14:26:16.501614 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:16.501693 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
W20260812 14:26:16.502307 25516 heartbeater.cc:644] Failed to heartbeat to 127.5.148.62:33463 (0 consecutive failures): Network error: Failed to ping master at 127.5.148.62:33463: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
I20260812 14:26:16.502624 25518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:16.502531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:34759)
0812 14:26:16.502579 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:16.502607 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 14:26:16.502914 25518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:16.502840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:39603)
0812 14:26:16.502878 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:16.502900 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 14:26:16.503033 25516 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 14:26:16.507983 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 25353
I20260812 14:26:16.508358 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:41091
--local_ip_for_outbound_sockets=127.5.148.3
--tserver_master_addrs=127.5.148.62:33463
--webserver_port=40453
--webserver_interface=127.5.148.3
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:16.595047 25520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:16.595268 25520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:16.595293 25520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:16.596555 25520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:16.596599 25520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:16.596628 25520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:26:16.598065 25520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:41091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=40453
--tserver_master_addrs=127.5.148.62:33463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:16.598256 25520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:16.598453 25520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:26:16.603746 25520 server_base.cc:1034] running on GCE node
W20260812 14:26:16.603681 25526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:16.603710 25527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:16.603703 25529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:16.604171 25520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:16.604473 25520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:16.605623 25520 hybrid_clock.cc:648] HybridClock initialized: now 1786544776605604 us; error 35 us; skew 500 ppm
Aug 12 14:26:16 dist-test-slave-8w9v krb5kdc[23774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544776, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:16.608290 25520 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:26:16.608808 25520 webserver.cc:460] Webserver started at http://127.5.148.3:40453/ using document root <none> and password file <none>
I20260812 14:26:16.609007 25520 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:16.609051 25520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:16.610396 25520 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:16.610999 25538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:16.611209 25520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:16.611286 25520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "a09dfb1718a644909deaaae117d972d5"
format_stamp: "Formatted at 2026-08-12 14:25:34 on dist-test-slave-8w9v"
server_key: "9762d105a8c907c115cd186b4aa2a117"
server_key_iv: "5b5ec19cc5ea48696dbfa8952f95bd25"
server_key_version: "encryptionkey@0"
I20260812 14:26:16.611639 25520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:16.622898 25520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:16.624930 25520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:16.625056 25520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:16.625267 25520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:16.625615 25520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:16.625651 25520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:16.625674 25520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:16.625720 25520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:16.631963 25520 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:41091
I20260812 14:26:16.632035 25669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:41091 every 8 connection(s)
I20260812 14:26:16.632421 25520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:26:16.633567 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:16.633185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:48001)
0812 14:26:16.633342 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:16.633419 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
W20260812 14:26:16.634027 25670 heartbeater.cc:644] Failed to heartbeat to 127.5.148.62:33463 (0 consecutive failures): Network error: Failed to ping master at 127.5.148.62:33463: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
I20260812 14:26:16.634312 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:16.634223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:35861)
0812 14:26:16.634267 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:16.634294 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 14:26:16.634632 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:16.634553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:35731)
0812 14:26:16.634597 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:16.634619 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 14:26:16.634675 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 25520
W20260812 14:26:16.634749 25670 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 14:26:17.398587 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:17.398283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:37601)
0812 14:26:17.398466 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:17.398519 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":75,"threads_started":1}
I20260812 14:26:17.503720 25677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:17.503435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:55409)
0812 14:26:17.503604 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:17.503654 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":67,"threads_started":1}
I20260812 14:26:17.635419 25678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:17.635176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:55981)
0812 14:26:17.635308 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:17.635354 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260812 14:26:17.732712 25679 raft_consensus.cc:482] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:26:17.732883 25679 raft_consensus.cc:504] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:26:17.733259 25679 leader_election.cc:288] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091), d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2:41509)
Aug 12 14:26:17 dist-test-slave-8w9v krb5kdc[23774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544776, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.2@KRBTEST.COM
Aug 12 14:26:17 dist-test-slave-8w9v krb5kdc[23774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544776, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.3@KRBTEST.COM
I20260812 14:26:17.746888 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:17.733449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:41091 (local address 127.5.148.1:46005)
0812 14:26:17.733749 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:17.733761 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:17.733834 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:17.734728 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:17.734731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:17.735208 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:17.735485 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:17.735494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:17.736999 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:17.737001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:17.738173 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:17.738177 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:17.739206 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:17.739208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:17.739263 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:17.739811 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:26:17.739817 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:17.742230 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:17.744949 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:17.744960 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:17.744963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:17.745515 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:17.745814 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:17.745815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:17.745815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:17.745882 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:17.746316 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:17.746324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:17.746469 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 14:26:17.746582 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 14:26:17.746778 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":4}
I20260812 14:26:17.746901 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:17.733593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:46005 (local address 127.5.148.3:41091)
0812 14:26:17.733770 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:17.733784 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:17.733828 (+ 44us) server_negotiation.cc:407] Connection header received
0812 14:26:17.734489 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:17.734492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:17.734538 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:17.734695 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:17.735608 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:17.736824 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:17.738295 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:17.739117 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:17.739240 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:17.742367 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:17.742371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:17.742371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:17.742384 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:17.744819 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:17.745605 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:17.745609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:17.745609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:17.745745 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:17.745960 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:17.745961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:17.745962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:17.746237 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:17.746338 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:17.746639 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 14:26:17.746777 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260812 14:26:17.746994 25682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:17.733762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:34651 (local address 127.5.148.2:41509)
0812 14:26:17.733933 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:17.733946 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:17.733985 (+ 39us) server_negotiation.cc:407] Connection header received
0812 14:26:17.734675 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:17.734679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:17.734738 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:17.734912 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:17.736221 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:17.737337 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:17.738233 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:17.738788 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:17.738875 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:17.742355 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:17.742359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:17.742360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:17.742374 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:17.744946 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:17.745483 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:17.745487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:17.745487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:17.745623 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:17.745960 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:17.745961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:17.745961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:17.746310 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:17.746410 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:17.746720 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 14:26:17.746874 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260812 14:26:17.747560 25681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:17.733657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:41509 (local address 127.5.148.1:34651)
0812 14:26:17.733936 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:17.733944 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:17.733979 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:17.734891 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:17.734892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:17.735207 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:17.735485 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:17.735494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:17.737423 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:17.737424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:17.738142 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:17.738145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:17.738979 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:17.738981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:17.739092 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:17.739662 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:26:17.739673 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:17.742219 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:17.745053 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:17.745054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:17.745055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:17.745382 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:17.745722 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:17.745724 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:17.745724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:17.745875 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:17.746443 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:17.746445 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:17.746499 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 14:26:17.746583 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 14:26:17.747452 (+ 869us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":704,"thread_start_us":50,"threads_started":1}
W20260812 14:26:17.748255 25209 leader_election.cc:341] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091): Not found: Tablet not found: a4552c68ca1d4a968f704656d4f1d7a9
W20260812 14:26:17.748544 25213 leader_election.cc:341] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2:41509): Not found: Tablet not found: a4552c68ca1d4a968f704656d4f1d7a9
I20260812 14:26:17.748605 25213 leader_election.cc:302] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa4fbea27ce14ff58406d44e57b14293; no voters: a09dfb1718a644909deaaae117d972d5, d20d1274fcae4bd5a72e189cc1b75f45
I20260812 14:26:17.748744 25679 raft_consensus.cc:2726] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:26:18.399420 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:18.399175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:43599)
0812 14:26:18.399325 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:18.399368 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260812 14:26:18.504593 25686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:18.504374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:36255)
0812 14:26:18.504510 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:18.504548 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260812 14:26:18.636253 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:18.635962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:39665)
0812 14:26:18.636114 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:18.636208 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260812 14:26:19.400315 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:19.399995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:53245)
0812 14:26:19.400227 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:19.400268 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":75,"threads_started":1}
I20260812 14:26:19.486390 25691 raft_consensus.cc:482] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:26:19.486506 25691 raft_consensus.cc:504] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:26:19.486752 25691 leader_election.cc:288] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091), d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2:41509)
W20260812 14:26:19.487267 25213 leader_election.cc:341] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2:41509): Not found: Tablet not found: a4552c68ca1d4a968f704656d4f1d7a9
W20260812 14:26:19.487360 25209 leader_election.cc:341] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091): Not found: Tablet not found: a4552c68ca1d4a968f704656d4f1d7a9
I20260812 14:26:19.487475 25209 leader_election.cc:302] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa4fbea27ce14ff58406d44e57b14293; no voters: a09dfb1718a644909deaaae117d972d5, d20d1274fcae4bd5a72e189cc1b75f45
I20260812 14:26:19.487600 25691 raft_consensus.cc:2726] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:26:19.505303 25692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:19.505088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:41043)
0812 14:26:19.505219 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:19.505258 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260812 14:26:19.637053 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:19.636805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:43001)
0812 14:26:19.636969 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:19.637011 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260812 14:26:20.401079 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:20.400823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:36859)
0812 14:26:20.400977 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:20.401021 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260812 14:26:20.505957 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:20.505738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:41431)
0812 14:26:20.505872 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:20.505911 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260812 14:26:20.637850 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:20.637592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:42027)
0812 14:26:20.637747 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:20.637797 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
I20260812 14:26:21.016338 25698 raft_consensus.cc:482] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:26:21.016453 25698 raft_consensus.cc:504] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:26:21.016700 25698 leader_election.cc:288] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091), d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2:41509)
W20260812 14:26:21.017151 25209 leader_election.cc:341] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091): Not found: Tablet not found: a4552c68ca1d4a968f704656d4f1d7a9
W20260812 14:26:21.017261 25213 leader_election.cc:341] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2:41509): Not found: Tablet not found: a4552c68ca1d4a968f704656d4f1d7a9
I20260812 14:26:21.017383 25213 leader_election.cc:302] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa4fbea27ce14ff58406d44e57b14293; no voters: a09dfb1718a644909deaaae117d972d5, d20d1274fcae4bd5a72e189cc1b75f45
I20260812 14:26:21.017468 25698 raft_consensus.cc:2726] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:26:21.401962 25699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:21.401711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:40385)
0812 14:26:21.401845 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:21.401889 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260812 14:26:21.506680 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:21.506448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:57577)
0812 14:26:21.506600 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:21.506632 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260812 14:26:21.638695 25701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:21.638428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:45777)
0812 14:26:21.638600 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:21.638646 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":71,"threads_started":1}
I20260812 14:26:22.402806 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:22.402571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:49039)
0812 14:26:22.402711 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:22.402758 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260812 14:26:22.507531 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:22.507264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:55863)
0812 14:26:22.507440 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:22.507480 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
I20260812 14:26:22.639663 25705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:22.639382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:46349)
0812 14:26:22.639564 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:22.639614 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":112,"threads_started":1}
I20260812 14:26:23.403693 25707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:23.403435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:43523)
0812 14:26:23.403597 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:23.403643 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260812 14:26:23.508435 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:23.508068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:39401)
0812 14:26:23.508345 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:23.508388 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":136,"threads_started":1}
I20260812 14:26:23.640681 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:23.640362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:37965)
0812 14:26:23.640583 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:23.640631 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":90,"threads_started":1}
I20260812 14:26:24.404647 25712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:24.404345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:54249)
0812 14:26:24.404548 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:24.404596 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":102,"threads_started":1}
I20260812 14:26:24.509294 25713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:24.509022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:50749)
0812 14:26:24.509200 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:24.509245 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
I20260812 14:26:24.641546 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:24.641294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:51251)
0812 14:26:24.641451 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:24.641498 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260812 14:26:25.385519 25715 raft_consensus.cc:482] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:26:25.385630 25715 raft_consensus.cc:504] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:26:25.385902 25715 leader_election.cc:288] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091), d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2:41509)
W20260812 14:26:25.386310 25209 leader_election.cc:341] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091): Not found: Tablet not found: a4552c68ca1d4a968f704656d4f1d7a9
W20260812 14:26:25.386446 25213 leader_election.cc:341] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2:41509): Not found: Tablet not found: a4552c68ca1d4a968f704656d4f1d7a9
I20260812 14:26:25.386538 25213 leader_election.cc:302] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa4fbea27ce14ff58406d44e57b14293; no voters: a09dfb1718a644909deaaae117d972d5, d20d1274fcae4bd5a72e189cc1b75f45
I20260812 14:26:25.386633 25715 raft_consensus.cc:2726] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:26:25.405570 25716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:25.405317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:40019)
0812 14:26:25.405471 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:25.405522 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260812 14:26:25.510200 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:25.509896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:52387)
0812 14:26:25.510086 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:25.510132 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":85,"threads_started":1}
I20260812 14:26:25.642515 25719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:25.642192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:53285)
0812 14:26:25.642379 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:25.642448 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
I20260812 14:26:26.406478 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:26.406214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:58071)
0812 14:26:26.406372 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:26.406428 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":68,"threads_started":1}
I20260812 14:26:26.511006 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:26.510746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:50751)
0812 14:26:26.510906 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:26.510955 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20260812 14:26:26.643325 25722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:26.643095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:43815)
0812 14:26:26.643229 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:26.643276 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260812 14:26:27.407240 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:27.407016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:55151)
0812 14:26:27.407151 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:27.407192 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260812 14:26:27.511864 25724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:27.511599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:48685)
0812 14:26:27.511764 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:27.511812 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260812 14:26:27.644429 25725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:27.644082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:59159)
0812 14:26:27.644328 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:27.644376 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":171,"threads_started":1}
I20260812 14:26:28.408113 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:28.407855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:38415)
0812 14:26:28.408013 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:28.408061 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260812 14:26:28.512745 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:28.512525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:38463)
0812 14:26:28.512659 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:28.512696 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260812 14:26:28.645251 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:28.644998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:38901)
0812 14:26:28.645153 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:28.645200 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
I20260812 14:26:29.408967 25730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:29.408698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:60417)
0812 14:26:29.408858 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:29.408899 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260812 14:26:29.513567 25732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:29.513295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:37445)
0812 14:26:29.513474 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:29.513516 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":97,"threads_started":1}
I20260812 14:26:29.646082 25733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:29.645841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:59661)
0812 14:26:29.645984 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:29.646031 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260812 14:26:30.409862 25734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:30.409598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:35157)
0812 14:26:30.409762 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:30.409809 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
I20260812 14:26:30.514312 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:30.514069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:43829)
0812 14:26:30.514204 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:30.514243 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260812 14:26:30.647022 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:30.646673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:58357)
0812 14:26:30.646918 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:30.646968 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":73,"threads_started":1}
I20260812 14:26:31.310952 25739 raft_consensus.cc:482] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:26:31.311089 25739 raft_consensus.cc:504] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09dfb1718a644909deaaae117d972d5" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 41091 } } peers { permanent_uuid: "d20d1274fcae4bd5a72e189cc1b75f45" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41509 } } peers { permanent_uuid: "fa4fbea27ce14ff58406d44e57b14293" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 33989 } }
I20260812 14:26:31.311332 25739 leader_election.cc:288] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091), d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2:41509)
W20260812 14:26:31.311769 25213 leader_election.cc:341] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20d1274fcae4bd5a72e189cc1b75f45 (127.5.148.2:41509): Not found: Tablet not found: a4552c68ca1d4a968f704656d4f1d7a9
W20260812 14:26:31.311910 25209 leader_election.cc:341] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a09dfb1718a644909deaaae117d972d5 (127.5.148.3:41091): Not found: Tablet not found: a4552c68ca1d4a968f704656d4f1d7a9
I20260812 14:26:31.312016 25209 leader_election.cc:302] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa4fbea27ce14ff58406d44e57b14293; no voters: a09dfb1718a644909deaaae117d972d5, d20d1274fcae4bd5a72e189cc1b75f45
I20260812 14:26:31.312110 25739 raft_consensus.cc:2726] T a4552c68ca1d4a968f704656d4f1d7a9 P fa4fbea27ce14ff58406d44e57b14293 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:26:31.410986 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:31.410691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.1:42591)
0812 14:26:31.410886 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:31.410936 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20260812 14:26:31.515375 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:31.514974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.2:54091)
0812 14:26:31.515242 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:31.515292 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":93,"threads_started":1}
I20260812 14:26:31.639602 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 25147
I20260812 14:26:31.647979 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:31.647730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33463 (local address 127.5.148.3:56301)
0812 14:26:31.647882 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:31.647928 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.62:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260812 14:26:31.649197 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 25353
I20260812 14:26:31.657541 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 25520
2026-08-12T14:26:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58827 ms)
[----------] 30 tests from SecurityITest (244358 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:31 dist-test-slave-8w9v krb5kdc[25746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:31 dist-test-slave-8w9v krb5kdc[25746](info): set up 2 sockets
Aug 12 14:26:31 dist-test-slave-8w9v krb5kdc[25746](info): commencing operation
krb5kdc: starting...
W20260812 14:26:32.807229 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:32 dist-test-slave-8w9v krb5kdc[25746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:26:32.943447 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:36965
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:36965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:33.028856 25763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:33.029143 25763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:33.029186 25763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:33.030396 25763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:33.030490 25763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:33.030516 25763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:33.030548 25763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:33.030579 25763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:33.032174 25763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:36965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:36965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:33.032498 25763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:33.032763 25763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:33.038612 25770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:33.038612 25772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:33.038683 25763 server_base.cc:1034] running on GCE node
W20260812 14:26:33.038614 25769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:33.039206 25763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:33.039557 25763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:33.040746 25763 hybrid_clock.cc:648] HybridClock initialized: now 1786544793040721 us; error 47 us; skew 500 ppm
Aug 12 14:26:33 dist-test-slave-8w9v krb5kdc[25746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544793, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:33.048311 25763 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:26:33.048758 25763 webserver.cc:460] Webserver started at http://127.5.148.62:33143/ using document root <none> and password file <none>
I20260812 14:26:33.048955 25763 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:33.048998 25763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:33.049093 25763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:33.050232 25763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "ec31f2a413d44178bb839ecc1833fba0"
format_stamp: "Formatted at 2026-08-12 14:26:33 on dist-test-slave-8w9v"
server_key: "6bb4abf0a6d131c87466297f9b387358"
server_key_iv: "447c4d4ec3437a48fddbf039661c6a25"
server_key_version: "encryptionkey@0"
I20260812 14:26:33.050554 25763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "ec31f2a413d44178bb839ecc1833fba0"
format_stamp: "Formatted at 2026-08-12 14:26:33 on dist-test-slave-8w9v"
server_key: "6bb4abf0a6d131c87466297f9b387358"
server_key_iv: "447c4d4ec3437a48fddbf039661c6a25"
server_key_version: "encryptionkey@0"
I20260812 14:26:33.051888 25763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:26:33.052717 25779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:33.052903 25763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:26:33.053025 25763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "ec31f2a413d44178bb839ecc1833fba0"
format_stamp: "Formatted at 2026-08-12 14:26:33 on dist-test-slave-8w9v"
server_key: "6bb4abf0a6d131c87466297f9b387358"
server_key_iv: "447c4d4ec3437a48fddbf039661c6a25"
server_key_version: "encryptionkey@0"
I20260812 14:26:33.053107 25763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:33.069516 25763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:33.071707 25763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:33.071887 25763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:33.076225 25763 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:36965
I20260812 14:26:33.076223 25839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:36965 every 8 connection(s)
I20260812 14:26:33.076699 25763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:33.077622 25840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:33.079860 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 25763
I20260812 14:26:33.079991 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:33.080195 25840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0: Bootstrap starting.
I20260812 14:26:33.080235 5712 external_mini_cluster.cc:1428] Setting key 419e81da8cfb1be25e4c0355b1125972
I20260812 14:26:33.080948 25840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:33.081297 25840 log.cc:824] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:33.082113 25840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0: No bootstrap required, opened a new log
I20260812 14:26:33.083979 25840 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec31f2a413d44178bb839ecc1833fba0" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 36965 } }
I20260812 14:26:33.084113 25840 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:33.084210 25840 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec31f2a413d44178bb839ecc1833fba0, State: Initialized, Role: FOLLOWER
I20260812 14:26:33.084353 25840 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec31f2a413d44178bb839ecc1833fba0" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 36965 } }
I20260812 14:26:33.084456 25840 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:33.084504 25840 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:33.084556 25840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:33.085147 25840 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec31f2a413d44178bb839ecc1833fba0" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 36965 } }
I20260812 14:26:33.085274 25840 leader_election.cc:302] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec31f2a413d44178bb839ecc1833fba0; no voters:
I20260812 14:26:33.085472 25840 leader_election.cc:288] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:33.085505 25845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:33.085654 25845 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [term 1 LEADER]: Becoming Leader. State: Replica: ec31f2a413d44178bb839ecc1833fba0, State: Running, Role: LEADER
I20260812 14:26:33.085754 25845 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec31f2a413d44178bb839ecc1833fba0" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 36965 } }
I20260812 14:26:33.085801 25840 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:33.086123 25845 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec31f2a413d44178bb839ecc1833fba0. Latest consensus state: current_term: 1 leader_uuid: "ec31f2a413d44178bb839ecc1833fba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec31f2a413d44178bb839ecc1833fba0" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 36965 } } }
I20260812 14:26:33.086197 25845 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:33.086877 25846 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec31f2a413d44178bb839ecc1833fba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec31f2a413d44178bb839ecc1833fba0" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 36965 } } }
I20260812 14:26:33.086982 25846 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:33.087168 25850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 12 14:26:33 dist-test-slave-8w9v krb5kdc[25746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:26:33.087738 25850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:33.089709 25850 catalog_manager.cc:1347] Generated new cluster ID: b9f31875732749deaeb5ccdd93f1af7c
I20260812 14:26:33.089782 25850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:33.092393 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.080803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46196 (local address 127.5.148.62:36965)
0812 14:26:33.080994 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:33.081024 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:33.081080 (+ 56us) server_negotiation.cc:407] Connection header received
0812 14:26:33.081695 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:33.081706 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:33.081775 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:33.081966 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:33.082692 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.083589 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.084561 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.085059 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.085221 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.088416 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:33.088420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:33.088421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:33.088436 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:33.090621 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:33.091033 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:33.091037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:33.091037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:33.091155 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:33.091368 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:33.091369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:33.091369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:33.091697 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:33.091868 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:33.091971 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 14:26:33.092156 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260812 14:26:33.097488 25850 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:33.098133 25850 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:33.101619 25850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0: Generated new TSK 0
I20260812 14:26:33.101802 25850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:26:33.134953 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:36965
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:33.221257 25868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:33.221477 25868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:33.221503 25868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:33.222687 25868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:33.222735 25868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:33.222766 25868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:26:33.224270 25868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:36965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:33.224524 25868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:33.224792 25868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:33.230106 25879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:33.230099 25876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:33.230113 25877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:33.230509 25868 server_base.cc:1034] running on GCE node
I20260812 14:26:33.230746 25868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:33.230997 25868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:33.232194 25868 hybrid_clock.cc:648] HybridClock initialized: now 1786544793232154 us; error 67 us; skew 500 ppm
Aug 12 14:26:33 dist-test-slave-8w9v krb5kdc[25746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544793, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:33.234851 25868 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:26:33.235325 25868 webserver.cc:460] Webserver started at http://127.5.148.1:34145/ using document root <none> and password file <none>
I20260812 14:26:33.235515 25868 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:33.235555 25868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:33.235652 25868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:33.236761 25868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "c9890ba5ffb24c6c91a7faf4f80118cb"
format_stamp: "Formatted at 2026-08-12 14:26:33 on dist-test-slave-8w9v"
server_key: "f312bda0ec7fba4953464c939f63a348"
server_key_iv: "0197ca58aaa64d0c33d87eee64d61f74"
server_key_version: "encryptionkey@0"
I20260812 14:26:33.237085 25868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "c9890ba5ffb24c6c91a7faf4f80118cb"
format_stamp: "Formatted at 2026-08-12 14:26:33 on dist-test-slave-8w9v"
server_key: "f312bda0ec7fba4953464c939f63a348"
server_key_iv: "0197ca58aaa64d0c33d87eee64d61f74"
server_key_version: "encryptionkey@0"
I20260812 14:26:33.238381 25868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:26:33.239108 25889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:33.239328 25868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:33.239426 25868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "c9890ba5ffb24c6c91a7faf4f80118cb"
format_stamp: "Formatted at 2026-08-12 14:26:33 on dist-test-slave-8w9v"
server_key: "f312bda0ec7fba4953464c939f63a348"
server_key_iv: "0197ca58aaa64d0c33d87eee64d61f74"
server_key_version: "encryptionkey@0"
I20260812 14:26:33.239483 25868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:33.252875 25868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:33.254844 25868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:33.254963 25868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:33.255153 25868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:33.255505 25868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:33.255541 25868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:33.255564 25868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:33.255576 25868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:33.261672 25868 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:40001
I20260812 14:26:33.261756 26022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:40001 every 8 connection(s)
I20260812 14:26:33.262117 25868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:26:33 dist-test-slave-8w9v krb5kdc[25746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544793, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:26:33.271340 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 25868
I20260812 14:26:33.271478 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:26:33.271715 5712 external_mini_cluster.cc:1428] Setting key d938978ac6559063796c66b9b5498962
I20260812 14:26:33.272053 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.262946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:42565 (local address 127.5.148.62:36965)
0812 14:26:33.262990 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:33.262992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:33.263219 (+ 227us) server_negotiation.cc:407] Connection header received
0812 14:26:33.263699 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:33.263700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:33.263710 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:33.263771 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:33.264795 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.265297 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.266295 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.266502 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.266554 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.268946 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:33.268948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:33.268948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:33.268955 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:33.270310 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:33.270815 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:33.270816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:33.270816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:33.270867 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:33.271230 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:33.271231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:33.271231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:33.271381 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:33.271463 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:33.271845 (+ 382us) server_negotiation.cc:299] Negotiation successful
0812 14:26:33.271953 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:26:33.272389 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.262902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:36965 (local address 127.5.148.1:42565)
0812 14:26:33.263080 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:33.263123 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:33.263617 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:33.263821 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:33.263824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:33.264500 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:33.264698 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.264703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.265412 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.265412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.266217 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.266219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.266614 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.266615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.266743 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.267275 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:26:33.267284 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:33.268833 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:33.270481 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:33.270488 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:33.270490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:33.270737 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:33.270979 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:33.270980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:33.270981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:33.271137 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:33.271516 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:33.271525 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:33.271669 (+ 144us) client_negotiation.cc:769] Sending connection context
0812 14:26:33.271805 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 14:26:33.272032 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260812 14:26:33.272939 26024 heartbeater.cc:347] Connected to a master server at 127.5.148.62:36965
I20260812 14:26:33.273052 26024 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:33.273252 26024 heartbeater.cc:511] Master 127.5.148.62:36965 requested a full tablet report, sending...
I20260812 14:26:33.273824 25798 ts_manager.cc:194] Registered new tserver with Master: c9890ba5ffb24c6c91a7faf4f80118cb (127.5.148.1:40001)
I20260812 14:26:33.274480 25798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:42565
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:26:33.303448 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:36965
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:33.389535 26030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:33.389750 26030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:33.389775 26030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:33.390869 26030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:33.390902 26030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:33.390928 26030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:26:33.392428 26030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:36965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:33.392625 26030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:33.392809 26030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:33.398304 26039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:33.398312 26041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:33.398375 26030 server_base.cc:1034] running on GCE node
W20260812 14:26:33.398362 26043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:33.398761 26030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:33.399055 26030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:33.400203 26030 hybrid_clock.cc:648] HybridClock initialized: now 1786544793400203 us; error 41 us; skew 500 ppm
Aug 12 14:26:33 dist-test-slave-8w9v krb5kdc[25746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544793, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:33.403067 26030 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:26:33.403673 26030 webserver.cc:460] Webserver started at http://127.5.148.2:33595/ using document root <none> and password file <none>
I20260812 14:26:33.403950 26030 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:33.404006 26030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:33.404210 26030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:33.405396 26030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "93273cde450a4ad68d1c791690bc3717"
format_stamp: "Formatted at 2026-08-12 14:26:33 on dist-test-slave-8w9v"
server_key: "5dfa4dfbc9dacefbb63df9d1c60bb491"
server_key_iv: "3da2f1ec3e8ec664341dabf7615167af"
server_key_version: "encryptionkey@0"
I20260812 14:26:33.405826 26030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "93273cde450a4ad68d1c791690bc3717"
format_stamp: "Formatted at 2026-08-12 14:26:33 on dist-test-slave-8w9v"
server_key: "5dfa4dfbc9dacefbb63df9d1c60bb491"
server_key_iv: "3da2f1ec3e8ec664341dabf7615167af"
server_key_version: "encryptionkey@0"
I20260812 14:26:33.407351 26030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:26:33.408370 26052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:33.408601 26030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:33.408730 26030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "93273cde450a4ad68d1c791690bc3717"
format_stamp: "Formatted at 2026-08-12 14:26:33 on dist-test-slave-8w9v"
server_key: "5dfa4dfbc9dacefbb63df9d1c60bb491"
server_key_iv: "3da2f1ec3e8ec664341dabf7615167af"
server_key_version: "encryptionkey@0"
I20260812 14:26:33.408816 26030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:33.422573 26030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:33.425359 26030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:33.425572 26030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:33.425861 26030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:33.426240 26030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:33.426303 26030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:33.426347 26030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:33.426390 26030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:33.433578 26030 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:39849
I20260812 14:26:33.433693 26181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:39849 every 8 connection(s)
I20260812 14:26:33.434160 26030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:26:33.440027 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 26030
I20260812 14:26:33.440173 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:26:33.440421 5712 external_mini_cluster.cc:1428] Setting key 77d067d1e3f0e4d19c17d3fbec219ebb
Aug 12 14:26:33 dist-test-slave-8w9v krb5kdc[25746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544793, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:26:33.445652 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.435098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:54625 (local address 127.5.148.62:36965)
0812 14:26:33.435160 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:33.435162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:33.435424 (+ 262us) server_negotiation.cc:407] Connection header received
0812 14:26:33.436020 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:33.436022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:33.436038 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:33.436157 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:33.437228 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.438102 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.439121 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.439485 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.439568 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.442143 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:33.442144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:33.442145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:33.442153 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:33.443953 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:33.444543 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:33.444545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:33.444546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:33.444615 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:33.444969 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:33.444970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:33.444970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:33.445091 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:33.445144 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:33.445514 (+ 370us) server_negotiation.cc:299] Negotiation successful
0812 14:26:33.445586 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:26:33.445959 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.435065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:36965 (local address 127.5.148.2:54625)
0812 14:26:33.435269 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:33.435334 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:33.435890 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:33.436238 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:33.436242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:33.436901 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:33.437116 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.437121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.438211 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.438211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.439027 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.439030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.439579 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.439580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.439685 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.440342 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:26:33.440351 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:33.442039 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:33.444053 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:33.444063 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:33.444065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:33.444444 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:33.444703 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:33.444704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:33.444704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:33.444876 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:33.445162 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:33.445170 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:33.445350 (+ 180us) client_negotiation.cc:769] Sending connection context
0812 14:26:33.445472 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 14:26:33.445670 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":121,"threads_started":1}
I20260812 14:26:33.446521 26182 heartbeater.cc:347] Connected to a master server at 127.5.148.62:36965
I20260812 14:26:33.446645 26182 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:33.446834 26182 heartbeater.cc:511] Master 127.5.148.62:36965 requested a full tablet report, sending...
I20260812 14:26:33.447317 25798 ts_manager.cc:194] Registered new tserver with Master: 93273cde450a4ad68d1c791690bc3717 (127.5.148.2:39849)
I20260812 14:26:33.447714 25798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:54625
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:26:33.471582 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:36965
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:33.566865 26189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:33.567092 26189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:33.567117 26189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:33.568323 26189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:33.568369 26189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:33.568396 26189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:26:33.569845 26189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:36965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:33.570053 26189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:33.570302 26189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:33.575608 26198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:33.575663 26196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:33.575615 26194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:33.575977 26189 server_base.cc:1034] running on GCE node
I20260812 14:26:33.576244 26189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:33.576496 26189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:33.577659 26189 hybrid_clock.cc:648] HybridClock initialized: now 1786544793577647 us; error 30 us; skew 500 ppm
Aug 12 14:26:33 dist-test-slave-8w9v krb5kdc[25746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544793, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:33.580374 26189 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:26:33.580806 26189 webserver.cc:460] Webserver started at http://127.5.148.3:42839/ using document root <none> and password file <none>
I20260812 14:26:33.581063 26189 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:33.581135 26189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:33.581278 26189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:33.582463 26189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f"
format_stamp: "Formatted at 2026-08-12 14:26:33 on dist-test-slave-8w9v"
server_key: "2dbf8fd4952f65ab9c1cb9957fc252e0"
server_key_iv: "08d8a377b6de34485c21f08b589df35c"
server_key_version: "encryptionkey@0"
I20260812 14:26:33.582852 26189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f"
format_stamp: "Formatted at 2026-08-12 14:26:33 on dist-test-slave-8w9v"
server_key: "2dbf8fd4952f65ab9c1cb9957fc252e0"
server_key_iv: "08d8a377b6de34485c21f08b589df35c"
server_key_version: "encryptionkey@0"
I20260812 14:26:33.584460 26189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260812 14:26:33.585270 26208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:33.585471 26189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:33.585590 26189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f"
format_stamp: "Formatted at 2026-08-12 14:26:33 on dist-test-slave-8w9v"
server_key: "2dbf8fd4952f65ab9c1cb9957fc252e0"
server_key_iv: "08d8a377b6de34485c21f08b589df35c"
server_key_version: "encryptionkey@0"
I20260812 14:26:33.585671 26189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:33.607844 26189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:33.610400 26189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:33.610586 26189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:33.610857 26189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:33.611212 26189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:33.611255 26189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:33.611280 26189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:33.611327 26189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:33.617743 26189 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:35057
I20260812 14:26:33.617822 26342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:35057 every 8 connection(s)
I20260812 14:26:33.618170 26189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:26:33.622303 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 26189
I20260812 14:26:33.622427 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:26:33.622619 5712 external_mini_cluster.cc:1428] Setting key 0795a5febf054f81b63693bf55e878ca
Aug 12 14:26:33 dist-test-slave-8w9v krb5kdc[25746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544793, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:26:33.628388 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.618996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:46601 (local address 127.5.148.62:36965)
0812 14:26:33.619060 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:33.619079 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:33.619273 (+ 194us) server_negotiation.cc:407] Connection header received
0812 14:26:33.619864 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:33.619865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:33.619880 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:33.619951 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:33.621095 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.621606 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.623096 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.623386 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.623486 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.625749 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:33.625751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:33.625751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:33.625758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:33.627080 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:33.627507 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:33.627508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:33.627508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:33.627562 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:33.627791 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:33.627792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:33.627792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:33.627893 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:33.627943 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:33.628255 (+ 312us) server_negotiation.cc:299] Negotiation successful
0812 14:26:33.628323 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:26:33.628667 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.618945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:36965 (local address 127.5.148.3:46601)
0812 14:26:33.619152 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:33.619189 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:33.619738 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:33.620033 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:33.620037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:33.620732 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:33.620992 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.620998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.621741 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.621742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.622983 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.622985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.623514 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.623515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.623695 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.624245 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:26:33.624253 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:33.625663 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:33.627173 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:33.627180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:33.627182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:33.627434 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:33.627623 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:33.627624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:33.627624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:33.627728 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:33.627946 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:33.627950 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:33.628072 (+ 122us) client_negotiation.cc:769] Sending connection context
0812 14:26:33.628229 (+ 157us) client_negotiation.cc:240] Negotiation successful
0812 14:26:33.628389 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":103,"threads_started":1}
I20260812 14:26:33.629253 26343 heartbeater.cc:347] Connected to a master server at 127.5.148.62:36965
I20260812 14:26:33.629384 26343 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:33.629630 26343 heartbeater.cc:511] Master 127.5.148.62:36965 requested a full tablet report, sending...
I20260812 14:26:33.630062 25798 ts_manager.cc:194] Registered new tserver with Master: 5dfe8207dc9945a9a9e9d5163dff2e5f (127.5.148.3:35057)
I20260812 14:26:33.630409 25798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:46601
I20260812 14:26:33.634441 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:26:33.641532 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.635853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.5.148.62:36965)
0812 14:26:33.635894 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:33.635896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:33.636029 (+ 133us) server_negotiation.cc:407] Connection header received
0812 14:26:33.636070 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:33.636071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:33.636080 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:33.636183 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:33.636730 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.637194 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.637868 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.638045 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.638092 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.639188 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:33.639189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:33.639190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:33.639196 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:33.640618 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:33.640910 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:33.640911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:33.640911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:33.640965 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:33.641110 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:33.641111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:33.641111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:33.641237 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:33.641307 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:33.641397 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 14:26:33.641470 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:26:33.643095 25798 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:26:33.644821 25798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:26:33.651532 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.647147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:35057 (local address 127.0.0.1:57540)
0812 14:26:33.647286 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:33.647312 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:33.647389 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:33.647976 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:33.647981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:33.647985 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:33.648054 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.648056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.649161 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.649162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.650353 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.650356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.651254 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.651255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.651310 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.651318 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 14:26:33.651400 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 14:26:33.651460 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260812 14:26:33.651546 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.647526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.5.148.3:35057)
0812 14:26:33.647713 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:33.647718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:33.647729 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:26:33.647771 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:33.647773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:33.647789 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:33.647883 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:33.648144 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.649056 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.650468 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.651129 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.651166 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.651204 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:33.651421 (+ 217us) server_negotiation.cc:299] Negotiation successful
0812 14:26:33.651460 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
I20260812 14:26:33.651901 26360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.647359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:40001 (local address 127.0.0.1:49502)
0812 14:26:33.647991 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:33.648003 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:33.648051 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:33.648515 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:33.648516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:33.648518 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:33.648560 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.648561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.649760 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.649761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.650653 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.650656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.651637 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.651638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.651713 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.651718 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:26:33.651766 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:26:33.651814 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260812 14:26:33.652034 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.647144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:39849 (local address 127.0.0.1:41188)
0812 14:26:33.647261 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:33.647300 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:33.647361 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:33.648363 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:33.648363 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:33.648365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:33.648411 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.648432 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.649704 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.649706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.650830 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.650833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.651737 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.651738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.651802 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.651807 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:26:33.651864 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 14:26:33.651905 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":80,"spinlock_wait_cycles":384,"thread_start_us":103,"threads_started":1}
I20260812 14:26:33.652292 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.647522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49502 (local address 127.5.148.1:40001)
0812 14:26:33.648032 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:33.648036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:33.648048 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:26:33.648252 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:33.648254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:33.648269 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:33.648349 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:33.648632 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.649679 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.650754 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.651469 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.651541 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.651594 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:33.652016 (+ 422us) server_negotiation.cc:299] Negotiation successful
0812 14:26:33.652068 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":108,"threads_started":1}
I20260812 14:26:33.652668 26269 tablet_service.cc:1467] Processing CreateTablet for tablet cf89db4cd2384f39be4df9ec299f236c (DEFAULT_TABLE table=test-table [id=9c805ab1221b4263895911b71d0ee11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:33.653024 26269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf89db4cd2384f39be4df9ec299f236c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:33.653002 25947 tablet_service.cc:1467] Processing CreateTablet for tablet cf89db4cd2384f39be4df9ec299f236c (DEFAULT_TABLE table=test-table [id=9c805ab1221b4263895911b71d0ee11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:33.653332 25947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf89db4cd2384f39be4df9ec299f236c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:33.655824 26365 tablet_bootstrap.cc:492] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f: Bootstrap starting.
I20260812 14:26:33.655954 26364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.647761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.5.148.2:39849)
0812 14:26:33.648199 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:33.648203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:33.648214 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:26:33.648251 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:33.648253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:33.648269 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:33.648350 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:33.648501 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.649602 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.650921 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.651590 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.651710 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.651772 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:33.651874 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 14:26:33.655851 (+ 3977us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":115,"threads_started":1}
I20260812 14:26:33.656638 26365 tablet_bootstrap.cc:654] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:33.656750 26110 tablet_service.cc:1467] Processing CreateTablet for tablet cf89db4cd2384f39be4df9ec299f236c (DEFAULT_TABLE table=test-table [id=9c805ab1221b4263895911b71d0ee11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:33.657086 26110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf89db4cd2384f39be4df9ec299f236c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:33.657253 26365 log.cc:824] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:33.657614 26366 tablet_bootstrap.cc:492] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb: Bootstrap starting.
I20260812 14:26:33.657985 26365 tablet_bootstrap.cc:492] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f: No bootstrap required, opened a new log
I20260812 14:26:33.658041 26365 ts_tablet_manager.cc:1397] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:26:33.659636 26365 raft_consensus.cc:348] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93273cde450a4ad68d1c791690bc3717" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39849 } } peers { permanent_uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35057 } } peers { permanent_uuid: "c9890ba5ffb24c6c91a7faf4f80118cb" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 40001 } }
I20260812 14:26:33.659801 26365 raft_consensus.cc:374] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:33.659866 26365 raft_consensus.cc:729] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dfe8207dc9945a9a9e9d5163dff2e5f, State: Initialized, Role: FOLLOWER
I20260812 14:26:33.659945 26370 tablet_bootstrap.cc:492] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717: Bootstrap starting.
I20260812 14:26:33.660003 26365 consensus_queue.cc:260] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93273cde450a4ad68d1c791690bc3717" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39849 } } peers { permanent_uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35057 } } peers { permanent_uuid: "c9890ba5ffb24c6c91a7faf4f80118cb" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 40001 } }
I20260812 14:26:33.660326 26365 ts_tablet_manager.cc:1428] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:26:33.660326 26343 heartbeater.cc:503] Master 127.5.148.62:36965 was elected leader, sending a full tablet report...
I20260812 14:26:33.660646 26370 tablet_bootstrap.cc:654] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:33.661055 26366 tablet_bootstrap.cc:654] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:33.661207 26370 log.cc:824] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:33.661674 26366 log.cc:824] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:33.662065 26370 tablet_bootstrap.cc:492] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717: No bootstrap required, opened a new log
I20260812 14:26:33.662119 26370 ts_tablet_manager.cc:1397] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:26:33.662477 26366 tablet_bootstrap.cc:492] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb: No bootstrap required, opened a new log
I20260812 14:26:33.662550 26366 ts_tablet_manager.cc:1397] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 14:26:33.664232 26370 raft_consensus.cc:348] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93273cde450a4ad68d1c791690bc3717" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39849 } } peers { permanent_uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35057 } } peers { permanent_uuid: "c9890ba5ffb24c6c91a7faf4f80118cb" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 40001 } }
I20260812 14:26:33.664412 26370 raft_consensus.cc:374] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:33.664454 26370 raft_consensus.cc:729] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93273cde450a4ad68d1c791690bc3717, State: Initialized, Role: FOLLOWER
I20260812 14:26:33.664595 26370 consensus_queue.cc:260] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93273cde450a4ad68d1c791690bc3717" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39849 } } peers { permanent_uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35057 } } peers { permanent_uuid: "c9890ba5ffb24c6c91a7faf4f80118cb" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 40001 } }
I20260812 14:26:33.664665 26366 raft_consensus.cc:348] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93273cde450a4ad68d1c791690bc3717" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39849 } } peers { permanent_uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35057 } } peers { permanent_uuid: "c9890ba5ffb24c6c91a7faf4f80118cb" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 40001 } }
I20260812 14:26:33.664834 26366 raft_consensus.cc:374] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:33.664898 26370 ts_tablet_manager.cc:1428] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:26:33.664940 26366 raft_consensus.cc:729] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9890ba5ffb24c6c91a7faf4f80118cb, State: Initialized, Role: FOLLOWER
I20260812 14:26:33.664940 26182 heartbeater.cc:503] Master 127.5.148.62:36965 was elected leader, sending a full tablet report...
I20260812 14:26:33.665089 26366 consensus_queue.cc:260] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93273cde450a4ad68d1c791690bc3717" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39849 } } peers { permanent_uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35057 } } peers { permanent_uuid: "c9890ba5ffb24c6c91a7faf4f80118cb" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 40001 } }
I20260812 14:26:33.665371 26366 ts_tablet_manager.cc:1428] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:26:33.665381 26024 heartbeater.cc:503] Master 127.5.148.62:36965 was elected leader, sending a full tablet report...
W20260812 14:26:33.684988 26183 tablet.cc:2369] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:26:33.758627 26371 raft_consensus.cc:482] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:26:33.758756 26371 raft_consensus.cc:504] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93273cde450a4ad68d1c791690bc3717" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39849 } } peers { permanent_uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35057 } } peers { permanent_uuid: "c9890ba5ffb24c6c91a7faf4f80118cb" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 40001 } }
I20260812 14:26:33.759177 26371 leader_election.cc:288] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93273cde450a4ad68d1c791690bc3717 (127.5.148.2:39849), c9890ba5ffb24c6c91a7faf4f80118cb (127.5.148.1:40001)
I20260812 14:26:33.762352 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.759476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:35147 (local address 127.5.148.1:40001)
0812 14:26:33.759531 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:33.759533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:33.759544 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:26:33.759598 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:33.759599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:33.759612 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:33.759692 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:33.760037 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.760568 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.761625 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.761998 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.762093 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.762107 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:33.762227 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 14:26:33.762274 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:26:33.762358 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.759390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:40001 (local address 127.5.148.3:35147)
0812 14:26:33.759458 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:33.759471 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:33.759521 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:33.759856 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:33.759857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:33.759858 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:33.759950 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.759952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.760668 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.760668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.761519 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.761522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.762078 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.762079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.762159 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.762164 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:26:33.762220 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 14:26:33.762271 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":60}
I20260812 14:26:33.762452 26364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.759644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:57963 (local address 127.5.148.2:39849)
0812 14:26:33.759714 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:33.759716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:33.759724 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:26:33.759750 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:33.759750 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:33.759760 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:33.759806 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:33.759978 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.760549 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.761627 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.762063 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.762131 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.762143 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:33.762325 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 14:26:33.762364 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:26:33.762677 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.759416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:39849 (local address 127.5.148.3:57963)
0812 14:26:33.759634 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:33.759645 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:33.759698 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:33.759825 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:33.759826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:33.759828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:33.759895 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.759897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.760628 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.760629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.761520 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:33.761523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.762175 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.762176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:33.762244 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.762247 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:26:33.762561 (+ 314us) client_negotiation.cc:240] Negotiation successful
0812 14:26:33.762606 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":57,"thread_start_us":140,"threads_started":1}
I20260812 14:26:33.762882 25970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf89db4cd2384f39be4df9ec299f236c" candidate_uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9890ba5ffb24c6c91a7faf4f80118cb" is_pre_election: true
I20260812 14:26:33.763065 25970 raft_consensus.cc:2454] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dfe8207dc9945a9a9e9d5163dff2e5f in term 0.
I20260812 14:26:33.763056 26132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf89db4cd2384f39be4df9ec299f236c" candidate_uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93273cde450a4ad68d1c791690bc3717" is_pre_election: true
I20260812 14:26:33.763229 26132 raft_consensus.cc:2454] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dfe8207dc9945a9a9e9d5163dff2e5f in term 0.
W20260812 14:26:33.763268 26025 tablet.cc:2369] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:26:33.763399 26213 leader_election.cc:302] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dfe8207dc9945a9a9e9d5163dff2e5f, c9890ba5ffb24c6c91a7faf4f80118cb; no voters:
I20260812 14:26:33.763603 26371 raft_consensus.cc:2781] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:26:33.763650 26371 raft_consensus.cc:482] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:26:33.763665 26371 raft_consensus.cc:3037] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:33.764559 26371 raft_consensus.cc:504] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93273cde450a4ad68d1c791690bc3717" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39849 } } peers { permanent_uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35057 } } peers { permanent_uuid: "c9890ba5ffb24c6c91a7faf4f80118cb" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 40001 } }
I20260812 14:26:33.764712 26371 leader_election.cc:288] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [CANDIDATE]: Term 1 election: Requested vote from peers 93273cde450a4ad68d1c791690bc3717 (127.5.148.2:39849), c9890ba5ffb24c6c91a7faf4f80118cb (127.5.148.1:40001)
I20260812 14:26:33.764928 26132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf89db4cd2384f39be4df9ec299f236c" candidate_uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93273cde450a4ad68d1c791690bc3717"
I20260812 14:26:33.764935 25970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf89db4cd2384f39be4df9ec299f236c" candidate_uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9890ba5ffb24c6c91a7faf4f80118cb"
I20260812 14:26:33.765017 25970 raft_consensus.cc:3037] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:33.765023 26132 raft_consensus.cc:3037] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:33.765838 25970 raft_consensus.cc:2454] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dfe8207dc9945a9a9e9d5163dff2e5f in term 1.
I20260812 14:26:33.765861 26132 raft_consensus.cc:2454] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dfe8207dc9945a9a9e9d5163dff2e5f in term 1.
I20260812 14:26:33.766086 26210 leader_election.cc:302] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dfe8207dc9945a9a9e9d5163dff2e5f, 93273cde450a4ad68d1c791690bc3717; no voters:
I20260812 14:26:33.766247 26371 raft_consensus.cc:2781] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:33.766479 26371 raft_consensus.cc:686] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [term 1 LEADER]: Becoming Leader. State: Replica: 5dfe8207dc9945a9a9e9d5163dff2e5f, State: Running, Role: LEADER
I20260812 14:26:33.766594 26371 consensus_queue.cc:237] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93273cde450a4ad68d1c791690bc3717" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39849 } } peers { permanent_uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35057 } } peers { permanent_uuid: "c9890ba5ffb24c6c91a7faf4f80118cb" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 40001 } }
I20260812 14:26:33.767455 25797 catalog_manager.cc:5495] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dfe8207dc9945a9a9e9d5163dff2e5f (127.5.148.3). New cstate: current_term: 1 leader_uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93273cde450a4ad68d1c791690bc3717" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dfe8207dc9945a9a9e9d5163dff2e5f" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 35057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9890ba5ffb24c6c91a7faf4f80118cb" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 40001 } health_report { overall_health: UNKNOWN } } }
I20260812 14:26:33.796434 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.793681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.5.148.3:35057)
0812 14:26:33.793733 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:33.793735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:33.793745 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:26:33.793847 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:33.793848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:33.793872 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:33.793964 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:26:33.794111 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.794641 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.795493 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:33.795704 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:33.795802 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:33.796000 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:26:33.796144 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:26:33.796267 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:33.796287 (+ 20us) server_negotiation.cc:299] Negotiation successful
0812 14:26:33.796311 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:26:33.798386 26132 raft_consensus.cc:1261] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717 [term 1 FOLLOWER]: Refusing update from remote peer 5dfe8207dc9945a9a9e9d5163dff2e5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:26:33.798386 25970 raft_consensus.cc:1261] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb [term 1 FOLLOWER]: Refusing update from remote peer 5dfe8207dc9945a9a9e9d5163dff2e5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:26:33.798831 26371 consensus_queue.cc:1035] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "93273cde450a4ad68d1c791690bc3717" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:26:33.798996 26377 consensus_queue.cc:1035] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9890ba5ffb24c6c91a7faf4f80118cb" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 40001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:26:33.804210 25798 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:26:33.804312 25798 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:26:33.805078 25798 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec31f2a413d44178bb839ecc1833fba0: Sending DeleteTablet for 3 replicas of tablet cf89db4cd2384f39be4df9ec299f236c
I20260812 14:26:33.805375 26110 tablet_service.cc:1514] Processing DeleteTablet for tablet cf89db4cd2384f39be4df9ec299f236c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:33 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:41188
I20260812 14:26:33.805416 26269 tablet_service.cc:1514] Processing DeleteTablet for tablet cf89db4cd2384f39be4df9ec299f236c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:33 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:57540
I20260812 14:26:33.805634 25947 tablet_service.cc:1514] Processing DeleteTablet for tablet cf89db4cd2384f39be4df9ec299f236c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:33 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:49502
I20260812 14:26:33.805848 26391 tablet_replica.cc:330] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb: stopping tablet replica
I20260812 14:26:33.805866 26390 tablet_replica.cc:330] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f: stopping tablet replica
I20260812 14:26:33.805878 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 25868
I20260812 14:26:33.805972 26390 raft_consensus.cc:2229] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:26:33.805994 26391 raft_consensus.cc:2229] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:26:33.806063 26390 raft_consensus.cc:2258] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:26:33.806095 26391 raft_consensus.cc:2258] T cf89db4cd2384f39be4df9ec299f236c P c9890ba5ffb24c6c91a7faf4f80118cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:26:33.808387 26389 tablet_replica.cc:330] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717: stopping tablet replica
I20260812 14:26:33.808511 26389 raft_consensus.cc:2229] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:26:33.808586 26389 raft_consensus.cc:2258] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:26:33.809096 26390 ts_tablet_manager.cc:1905] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:26:33.809095 26389 ts_tablet_manager.cc:1905] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:26:33.811024 26390 ts_tablet_manager.cc:1918] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:26:33.811041 26389 ts_tablet_manager.cc:1918] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:26:33.811107 26389 log.cc:1196] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/cf89db4cd2384f39be4df9ec299f236c
I20260812 14:26:33.811141 26390 log.cc:1196] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/cf89db4cd2384f39be4df9ec299f236c
I20260812 14:26:33.811444 26389 ts_tablet_manager.cc:1939] T cf89db4cd2384f39be4df9ec299f236c P 93273cde450a4ad68d1c791690bc3717: Deleting consensus metadata
I20260812 14:26:33.811493 26390 ts_tablet_manager.cc:1939] T cf89db4cd2384f39be4df9ec299f236c P 5dfe8207dc9945a9a9e9d5163dff2e5f: Deleting consensus metadata
I20260812 14:26:33.811883 25780 catalog_manager.cc:4841] TS 93273cde450a4ad68d1c791690bc3717 (127.5.148.2:39849): tablet cf89db4cd2384f39be4df9ec299f236c (table test-table [id=9c805ab1221b4263895911b71d0ee11d]) successfully deleted
I20260812 14:26:33.812284 25782 catalog_manager.cc:4841] TS 5dfe8207dc9945a9a9e9d5163dff2e5f (127.5.148.3:35057): tablet cf89db4cd2384f39be4df9ec299f236c (table test-table [id=9c805ab1221b4263895911b71d0ee11d]) successfully deleted
W20260812 14:26:33.816413 25782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:40001 (error 108)
I20260812 14:26:33.816730 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 26030
I20260812 14:26:33.818136 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:33.816825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:40001 (local address 127.0.0.1:49518)
0812 14:26:33.818049 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:33.818093 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:40001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1203}
W20260812 14:26:33.819005 25782 catalog_manager.cc:4584] TS c9890ba5ffb24c6c91a7faf4f80118cb (127.5.148.1:40001): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf89db4cd2384f39be4df9ec299f236c: Network error: Client connection negotiation failed: client connection to 127.5.148.1:40001: connect: Connection refused (error 111)
I20260812 14:26:33.824432 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 26189
I20260812 14:26:33.834798 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 25763
2026-08-12T14:26:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2178 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T14:26:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:33Z Disabled control of system clock
I20260812 14:26:33.868202 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:35611
--webserver_interface=127.5.148.62
--webserver_port=0
--builtin_ntp_servers=127.5.148.20:33159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:35611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:26:33.953604 26397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:33.953917 26397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:33.953953 26397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:33.955973 26397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:33.956059 26397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:33.956084 26397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:33.956110 26397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:33.956187 26397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:33.958505 26397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:33159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:35611
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:35611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:33.958813 26397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:33.959194 26397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:33.965301 26405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:33.965319 26403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:33.965525 26397 server_base.cc:1034] running on GCE node
W20260812 14:26:33.965294 26402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:33.965873 26397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:33.966153 26397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:33.967341 26397 hybrid_clock.cc:648] HybridClock initialized: now 1786544793967324 us; error 29 us; skew 500 ppm
I20260812 14:26:33.968758 26397 webserver.cc:460] Webserver started at http://127.5.148.62:45299/ using document root <none> and password file <none>
I20260812 14:26:33.969055 26397 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:33.969106 26397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:33.969280 26397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:33.970431 26397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "0d25a79b208d4d73bdff982dd3c478fc"
format_stamp: "Formatted at 2026-08-12 14:26:33 on dist-test-slave-8w9v"
server_key: "d946f5b00dc076322c84dd695bf3bf3d"
server_key_iv: "82aa0a3c84e769a0c846c4ebd0953928"
server_key_version: "encryptionkey@0"
I20260812 14:26:33.970805 26397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "0d25a79b208d4d73bdff982dd3c478fc"
format_stamp: "Formatted at 2026-08-12 14:26:33 on dist-test-slave-8w9v"
server_key: "d946f5b00dc076322c84dd695bf3bf3d"
server_key_iv: "82aa0a3c84e769a0c846c4ebd0953928"
server_key_version: "encryptionkey@0"
I20260812 14:26:33.972241 26397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260812 14:26:33.973047 26411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:33.973228 26397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:26:33.973327 26397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "0d25a79b208d4d73bdff982dd3c478fc"
format_stamp: "Formatted at 2026-08-12 14:26:33 on dist-test-slave-8w9v"
server_key: "d946f5b00dc076322c84dd695bf3bf3d"
server_key_iv: "82aa0a3c84e769a0c846c4ebd0953928"
server_key_version: "encryptionkey@0"
I20260812 14:26:33.973403 26397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:33.992923 26397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:33.993289 26397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:33.993456 26397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:33.997628 26397 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:35611
I20260812 14:26:33.997664 26472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:35611 every 8 connection(s)
I20260812 14:26:33.998035 26397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:33.999089 26473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:34.001508 26473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc: Bootstrap starting.
I20260812 14:26:34.002121 26473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:34.002442 26473 log.cc:824] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:34.003082 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 26397
I20260812 14:26:34.003204 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:34.003264 26473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc: No bootstrap required, opened a new log
I20260812 14:26:34.003413 5712 external_mini_cluster.cc:1428] Setting key f36cdf9a27ea5c1806aef74371d99517
I20260812 14:26:34.005254 26473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d25a79b208d4d73bdff982dd3c478fc" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35611 } }
I20260812 14:26:34.005431 26473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:34.005479 26473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d25a79b208d4d73bdff982dd3c478fc, State: Initialized, Role: FOLLOWER
I20260812 14:26:34.005611 26473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d25a79b208d4d73bdff982dd3c478fc" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35611 } }
I20260812 14:26:34.005702 26473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:34.005748 26473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:34.005820 26473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:34.006414 26473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d25a79b208d4d73bdff982dd3c478fc" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35611 } }
I20260812 14:26:34.006536 26473 leader_election.cc:302] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d25a79b208d4d73bdff982dd3c478fc; no voters:
I20260812 14:26:34.006734 26473 leader_election.cc:288] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:34.006768 26481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:34.006909 26481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [term 1 LEADER]: Becoming Leader. State: Replica: 0d25a79b208d4d73bdff982dd3c478fc, State: Running, Role: LEADER
I20260812 14:26:34.007020 26473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:34.006997 26481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d25a79b208d4d73bdff982dd3c478fc" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35611 } }
I20260812 14:26:34.007377 26481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d25a79b208d4d73bdff982dd3c478fc. Latest consensus state: current_term: 1 leader_uuid: "0d25a79b208d4d73bdff982dd3c478fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d25a79b208d4d73bdff982dd3c478fc" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35611 } } }
I20260812 14:26:34.007475 26481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:34.007390 26482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d25a79b208d4d73bdff982dd3c478fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d25a79b208d4d73bdff982dd3c478fc" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35611 } } }
I20260812 14:26:34.007638 26482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:34.007691 26490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:34.008302 26490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:34.010023 26490 catalog_manager.cc:1347] Generated new cluster ID: 0a2c8ceaecd54095bc68cee1cbefeb88
I20260812 14:26:34.010079 26490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:34.011129 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.003979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49306 (local address 127.5.148.62:35611)
0812 14:26:34.004266 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:34.004276 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:34.004308 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:26:34.005232 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:34.005239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:34.005305 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:34.005456 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:34.005676 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.007599 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.008813 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.009426 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.009588 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.010381 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:34.010385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:34.010386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:34.010598 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:34.010649 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:34.010664 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:34.010740 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:34.010820 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 14:26:34.010909 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
I20260812 14:26:34.014976 26490 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:34.015588 26490 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:34.020025 26490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc: Generated new TSK 0
I20260812 14:26:34.020247 26490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:34.028550 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35611
--builtin_ntp_servers=127.5.148.20:33159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:26:34.113989 26508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:34.114249 26508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:34.114290 26508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:34.115492 26508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:34.115557 26508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:34.115602 26508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:26:34.117111 26508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:33159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:34.117391 26508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:34.117717 26508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:34.123070 26513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:34.123144 26516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:34.123162 26508 server_base.cc:1034] running on GCE node
W20260812 14:26:34.123279 26514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:34.123512 26508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:34.123773 26508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:34.124955 26508 hybrid_clock.cc:648] HybridClock initialized: now 1786544794124947 us; error 36 us; skew 500 ppm
I20260812 14:26:34.126343 26508 webserver.cc:460] Webserver started at http://127.5.148.1:44269/ using document root <none> and password file <none>
I20260812 14:26:34.126621 26508 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:34.126693 26508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:34.126853 26508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:34.127995 26508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7"
format_stamp: "Formatted at 2026-08-12 14:26:34 on dist-test-slave-8w9v"
server_key: "00653140576114c92e2a70b25e9275a6"
server_key_iv: "98ed09f199e39ce415708613c86123f0"
server_key_version: "encryptionkey@0"
I20260812 14:26:34.128428 26508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7"
format_stamp: "Formatted at 2026-08-12 14:26:34 on dist-test-slave-8w9v"
server_key: "00653140576114c92e2a70b25e9275a6"
server_key_iv: "98ed09f199e39ce415708613c86123f0"
server_key_version: "encryptionkey@0"
I20260812 14:26:34.129886 26508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:34.130750 26524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:34.130944 26508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:34.131053 26508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7"
format_stamp: "Formatted at 2026-08-12 14:26:34 on dist-test-slave-8w9v"
server_key: "00653140576114c92e2a70b25e9275a6"
server_key_iv: "98ed09f199e39ce415708613c86123f0"
server_key_version: "encryptionkey@0"
I20260812 14:26:34.131134 26508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:34.138070 26508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:34.138376 26508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:34.138527 26508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:34.138751 26508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:34.139108 26508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:34.139173 26508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:34.139215 26508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:34.139243 26508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:34.145342 26508 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:38425
I20260812 14:26:34.145416 26673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:38425 every 8 connection(s)
I20260812 14:26:34.145767 26508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:26:34.151219 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.146536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:47227 (local address 127.5.148.62:35611)
0812 14:26:34.146594 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:34.146597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:34.146812 (+ 215us) server_negotiation.cc:407] Connection header received
0812 14:26:34.147380 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:34.147382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:34.147397 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:34.147458 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:34.148230 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.148748 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.149763 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.149974 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.150056 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.150908 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:34.150909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:34.150910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:34.150915 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:34.150931 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:34.150941 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:34.151004 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:34.151103 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 14:26:34.151135 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:26:34.151448 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.146490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35611 (local address 127.5.148.1:47227)
0812 14:26:34.146680 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:34.146716 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:34.147263 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:34.147545 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:34.147549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:34.147559 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:34.148091 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.148096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.148853 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.148854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.149661 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.149663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.150066 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.150067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.150211 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.150763 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:26:34.150763 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:34.150793 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:26:34.150815 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:34.151006 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:34.151013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:34.151016 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:26:34.151103 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 14:26:34.151210 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260812 14:26:34.151959 26674 heartbeater.cc:347] Connected to a master server at 127.5.148.62:35611
I20260812 14:26:34.152063 26674 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:34.152280 26674 heartbeater.cc:511] Master 127.5.148.62:35611 requested a full tablet report, sending...
I20260812 14:26:34.152760 26431 ts_manager.cc:194] Registered new tserver with Master: ba101b7fbdc0460db14a55dfb5e4fdd7 (127.5.148.1:38425)
I20260812 14:26:34.153407 26431 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.148.1:47227
I20260812 14:26:34.155229 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 26508
I20260812 14:26:34.155345 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:26:34.155576 5712 external_mini_cluster.cc:1428] Setting key 2a4f1b6a7d4b3ee304005a9874b85f8c
I20260812 14:26:34.156575 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35611
--builtin_ntp_servers=127.5.148.20:33159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:26:34.242682 26678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:34.242903 26678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:34.242926 26678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:34.244097 26678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:34.244187 26678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:34.244248 26678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:26:34.245654 26678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:33159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:34.245913 26678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:34.246248 26678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:34.251524 26687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:34.251564 26685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:34.251534 26683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:34.251713 26678 server_base.cc:1034] running on GCE node
I20260812 14:26:34.252002 26678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:34.252280 26678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:34.253454 26678 hybrid_clock.cc:648] HybridClock initialized: now 1786544794253454 us; error 41 us; skew 500 ppm
I20260812 14:26:34.254781 26678 webserver.cc:460] Webserver started at http://127.5.148.2:38265/ using document root <none> and password file <none>
I20260812 14:26:34.255045 26678 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:34.255116 26678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:34.255278 26678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:34.256528 26678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "d03178c76c8949b0b2c53dcde35cde7c"
format_stamp: "Formatted at 2026-08-12 14:26:34 on dist-test-slave-8w9v"
server_key: "488d255118c6ab5912cbea7508d42e31"
server_key_iv: "aa1c56b2cced52680b710a2ec057fcd6"
server_key_version: "encryptionkey@0"
I20260812 14:26:34.256929 26678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "d03178c76c8949b0b2c53dcde35cde7c"
format_stamp: "Formatted at 2026-08-12 14:26:34 on dist-test-slave-8w9v"
server_key: "488d255118c6ab5912cbea7508d42e31"
server_key_iv: "aa1c56b2cced52680b710a2ec057fcd6"
server_key_version: "encryptionkey@0"
I20260812 14:26:34.258297 26678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:26:34.259090 26693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:34.259294 26678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:34.259402 26678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "d03178c76c8949b0b2c53dcde35cde7c"
format_stamp: "Formatted at 2026-08-12 14:26:34 on dist-test-slave-8w9v"
server_key: "488d255118c6ab5912cbea7508d42e31"
server_key_iv: "aa1c56b2cced52680b710a2ec057fcd6"
server_key_version: "encryptionkey@0"
I20260812 14:26:34.259488 26678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:34.289949 26678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:34.290333 26678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:34.290510 26678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:34.290776 26678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:34.291165 26678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:34.291224 26678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:34.291272 26678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:34.291296 26678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:34.297459 26678 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:33607
I20260812 14:26:34.297513 26827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:33607 every 8 connection(s)
I20260812 14:26:34.297902 26678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:26:34.303519 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 26678
I20260812 14:26:34.303623 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.298793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:33945 (local address 127.5.148.62:35611)
0812 14:26:34.298840 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:34.298842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:34.299125 (+ 283us) server_negotiation.cc:407] Connection header received
0812 14:26:34.299623 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:34.299624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:34.299634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:34.299699 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:34.300560 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.301056 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.302090 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.302276 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.302330 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.303285 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:34.303287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:34.303287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:34.303293 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:34.303310 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:34.303315 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:34.303371 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:34.303468 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 14:26:34.303502 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:26:34.303661 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:26:34.303791 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.298755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35611 (local address 127.5.148.2:33945)
0812 14:26:34.298982 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:34.299027 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:34.299537 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:34.299788 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:34.299791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:34.299800 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:34.300454 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.300461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.301157 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.301158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.302010 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.302013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.302376 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.302377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.302495 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.303120 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:26:34.303121 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:34.303153 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:26:34.303177 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:34.303380 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:34.303385 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:34.303388 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:26:34.303444 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 14:26:34.303531 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":118,"threads_started":1}
I20260812 14:26:34.303900 5712 external_mini_cluster.cc:1428] Setting key 62a70f7b32ec817338e1c05f22fe041b
I20260812 14:26:34.304590 26828 heartbeater.cc:347] Connected to a master server at 127.5.148.62:35611
I20260812 14:26:34.304700 26828 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:34.304905 26828 heartbeater.cc:511] Master 127.5.148.62:35611 requested a full tablet report, sending...
I20260812 14:26:34.305090 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35611
--builtin_ntp_servers=127.5.148.20:33159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 14:26:34.305387 26431 ts_manager.cc:194] Registered new tserver with Master: d03178c76c8949b0b2c53dcde35cde7c (127.5.148.2:33607)
I20260812 14:26:34.305878 26431 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.148.2:33945
W20260812 14:26:34.390529 26831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:34.390815 26831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:34.390841 26831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:34.391964 26831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:34.391997 26831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:34.392021 26831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:26:34.393321 26831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:33159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:35611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:34.393504 26831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:34.393745 26831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:34.398944 26838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:34.398985 26840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:34.398937 26842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:34.399230 26831 server_base.cc:1034] running on GCE node
I20260812 14:26:34.399487 26831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:34.399744 26831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:34.400921 26831 hybrid_clock.cc:648] HybridClock initialized: now 1786544794400905 us; error 33 us; skew 500 ppm
I20260812 14:26:34.402360 26831 webserver.cc:460] Webserver started at http://127.5.148.3:35913/ using document root <none> and password file <none>
I20260812 14:26:34.402632 26831 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:34.402714 26831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:34.402881 26831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:34.404088 26831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "9efdae7898d949dbafafef8318194ef0"
format_stamp: "Formatted at 2026-08-12 14:26:34 on dist-test-slave-8w9v"
server_key: "95ecbba37a42b8c45c189eb8d98ab537"
server_key_iv: "2b37d86820083e07f6ce46b002ae9ba3"
server_key_version: "encryptionkey@0"
I20260812 14:26:34.404538 26831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "9efdae7898d949dbafafef8318194ef0"
format_stamp: "Formatted at 2026-08-12 14:26:34 on dist-test-slave-8w9v"
server_key: "95ecbba37a42b8c45c189eb8d98ab537"
server_key_iv: "2b37d86820083e07f6ce46b002ae9ba3"
server_key_version: "encryptionkey@0"
I20260812 14:26:34.405978 26831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:26:34.406776 26848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:34.406957 26831 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:26:34.407063 26831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "9efdae7898d949dbafafef8318194ef0"
format_stamp: "Formatted at 2026-08-12 14:26:34 on dist-test-slave-8w9v"
server_key: "95ecbba37a42b8c45c189eb8d98ab537"
server_key_iv: "2b37d86820083e07f6ce46b002ae9ba3"
server_key_version: "encryptionkey@0"
I20260812 14:26:34.407157 26831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:34.428483 26831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:34.428874 26831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:34.429062 26831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:34.429371 26831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:34.429785 26831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:34.429847 26831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:34.429898 26831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:34.429948 26831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:34.437167 26831 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:39563
I20260812 14:26:34.437222 26991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:39563 every 8 connection(s)
I20260812 14:26:34.437655 26831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:26:34.441568 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 26831
I20260812 14:26:34.441675 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:26:34.441898 5712 external_mini_cluster.cc:1428] Setting key bfc69189506892ee7632b492f3a09f1d
I20260812 14:26:34.443511 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.438510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:60693 (local address 127.5.148.62:35611)
0812 14:26:34.438559 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:34.438561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:34.438797 (+ 236us) server_negotiation.cc:407] Connection header received
0812 14:26:34.439374 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:34.439376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:34.439390 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:34.439456 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:34.440233 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.440747 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.441833 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.442118 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.442210 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.443187 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:34.443188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:34.443189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:34.443193 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:34.443208 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:34.443212 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:34.443280 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:34.443381 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 14:26:34.443429 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:26:34.443746 26996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.438468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:35611 (local address 127.5.148.3:60693)
0812 14:26:34.438660 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:34.438696 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:34.439255 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:34.439544 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:34.439548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:34.439566 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:34.440103 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.440108 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.440857 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.440859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.441731 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.441735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.442227 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.442228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.442387 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.443034 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:26:34.443035 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:34.443067 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:26:34.443091 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:34.443281 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:34.443288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:34.443292 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:26:34.443383 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 14:26:34.443498 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260812 14:26:34.444262 26992 heartbeater.cc:347] Connected to a master server at 127.5.148.62:35611
I20260812 14:26:34.444372 26992 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:34.444574 26992 heartbeater.cc:511] Master 127.5.148.62:35611 requested a full tablet report, sending...
I20260812 14:26:34.444962 26431 ts_manager.cc:194] Registered new tserver with Master: 9efdae7898d949dbafafef8318194ef0 (127.5.148.3:39563)
I20260812 14:26:34.445305 26431 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.148.3:60693
I20260812 14:26:34.453750 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:26:34.458356 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.455277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.5.148.62:35611)
0812 14:26:34.455326 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:34.455329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:34.455461 (+ 132us) server_negotiation.cc:407] Connection header received
0812 14:26:34.455537 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:34.455538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:34.455559 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:34.455626 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:34.455809 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.456370 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.457147 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.457336 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.457431 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.458087 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:34.458088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:34.458088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:34.458094 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:34.458108 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:34.458112 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:34.458178 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:34.458232 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 14:26:34.458261 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:26:34.460048 26431 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:26:34.461745 26431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:26:34.468401 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.464194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.5.148.1:38425)
0812 14:26:34.464574 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:34.464578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:34.464587 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:26:34.464674 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:34.464676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:34.464694 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:34.464770 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:34.465013 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.466198 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.467417 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.467939 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.468032 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.468135 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:34.468265 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 14:26:34.468304 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":89,"threads_started":1}
I20260812 14:26:34.468475 27009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.464311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:33607 (local address 127.0.0.1:53982)
0812 14:26:34.464454 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:34.464465 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:34.464522 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:34.464946 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:34.464948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:34.464950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:34.465000 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.465002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.465943 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.465944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.467186 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.467189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.468176 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.468177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.468238 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.468243 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:26:34.468276 (+ 33us) client_negotiation.cc:240] Negotiation successful
0812 14:26:34.468320 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":6,"thread_start_us":79,"threads_started":1}
I20260812 14:26:34.468396 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.464064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:38425 (local address 127.0.0.1:53434)
0812 14:26:34.464401 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:34.464412 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:34.464454 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:34.464862 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:34.464865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:34.464868 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:34.464945 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.464947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.466292 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.466293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.467333 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.467335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.468033 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.468034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.468115 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.468162 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 14:26:34.468238 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 14:26:34.468304 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":131,"threads_started":1}
I20260812 14:26:34.468681 27007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.464162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.5.148.3:39563)
0812 14:26:34.464818 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:34.464822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:34.464832 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:26:34.464871 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:34.464872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:34.464887 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:34.464970 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:34.465164 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.466250 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.467327 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.468045 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.468150 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.468244 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:34.468521 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 14:26:34.468572 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":102,"threads_started":1}
I20260812 14:26:34.468799 27004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.464061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:39563 (local address 127.0.0.1:60398)
0812 14:26:34.464282 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:34.464329 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:34.464401 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:34.465044 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:34.465045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:34.465046 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:34.465077 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.465078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.466335 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.466336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.467212 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.467215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.468278 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.468279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.468331 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.468333 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:26:34.468382 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:26:34.468435 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
I20260812 14:26:34.469408 26586 tablet_service.cc:1467] Processing CreateTablet for tablet b23c3520a56f4dafb4be78690e3f396d (DEFAULT_TABLE table=test-table [id=d75df6b6b28b48c38df5b004f96a575e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:34.469677 27011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.464536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.5.148.2:33607)
0812 14:26:34.464748 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:34.464754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:34.464767 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:26:34.464812 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:34.464814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:34.464831 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:34.465082 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:34.465147 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.465836 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.467283 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.468045 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.468439 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.469511 (+ 1072us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:34.469534 (+ 23us) server_negotiation.cc:299] Negotiation successful
0812 14:26:34.469569 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":139,"threads_started":1}
I20260812 14:26:34.469837 26586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23c3520a56f4dafb4be78690e3f396d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:34.469748 26911 tablet_service.cc:1467] Processing CreateTablet for tablet b23c3520a56f4dafb4be78690e3f396d (DEFAULT_TABLE table=test-table [id=d75df6b6b28b48c38df5b004f96a575e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:34.470175 26911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23c3520a56f4dafb4be78690e3f396d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:34.470263 26751 tablet_service.cc:1467] Processing CreateTablet for tablet b23c3520a56f4dafb4be78690e3f396d (DEFAULT_TABLE table=test-table [id=d75df6b6b28b48c38df5b004f96a575e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:34.470621 26751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23c3520a56f4dafb4be78690e3f396d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:34.473443 27012 tablet_bootstrap.cc:492] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c: Bootstrap starting.
I20260812 14:26:34.473698 27014 tablet_bootstrap.cc:492] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7: Bootstrap starting.
I20260812 14:26:34.474218 27012 tablet_bootstrap.cc:654] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:34.474274 27014 tablet_bootstrap.cc:654] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:34.474535 27012 log.cc:824] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:34.474570 27014 log.cc:824] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:34.474954 27013 tablet_bootstrap.cc:492] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0: Bootstrap starting.
I20260812 14:26:34.475445 27012 tablet_bootstrap.cc:492] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c: No bootstrap required, opened a new log
I20260812 14:26:34.475497 27014 tablet_bootstrap.cc:492] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7: No bootstrap required, opened a new log
I20260812 14:26:34.475521 27012 ts_tablet_manager.cc:1397] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:26:34.475559 27014 ts_tablet_manager.cc:1397] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:26:34.475632 27013 tablet_bootstrap.cc:654] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:34.476003 27013 log.cc:824] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:34.476889 27013 tablet_bootstrap.cc:492] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0: No bootstrap required, opened a new log
I20260812 14:26:34.476972 27013 ts_tablet_manager.cc:1397] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:26:34.477746 27012 raft_consensus.cc:348] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efdae7898d949dbafafef8318194ef0" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39563 } } peers { permanent_uuid: "d03178c76c8949b0b2c53dcde35cde7c" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33607 } } peers { permanent_uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 38425 } }
I20260812 14:26:34.477913 27012 raft_consensus.cc:374] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:34.477958 27012 raft_consensus.cc:729] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03178c76c8949b0b2c53dcde35cde7c, State: Initialized, Role: FOLLOWER
I20260812 14:26:34.478082 27012 consensus_queue.cc:260] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efdae7898d949dbafafef8318194ef0" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39563 } } peers { permanent_uuid: "d03178c76c8949b0b2c53dcde35cde7c" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33607 } } peers { permanent_uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 38425 } }
I20260812 14:26:34.478359 27012 ts_tablet_manager.cc:1428] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:26:34.478385 26828 heartbeater.cc:503] Master 127.5.148.62:35611 was elected leader, sending a full tablet report...
I20260812 14:26:34.478580 27014 raft_consensus.cc:348] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efdae7898d949dbafafef8318194ef0" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39563 } } peers { permanent_uuid: "d03178c76c8949b0b2c53dcde35cde7c" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33607 } } peers { permanent_uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 38425 } }
I20260812 14:26:34.478747 27014 raft_consensus.cc:374] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:34.478796 27014 raft_consensus.cc:729] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba101b7fbdc0460db14a55dfb5e4fdd7, State: Initialized, Role: FOLLOWER
I20260812 14:26:34.478920 27014 consensus_queue.cc:260] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efdae7898d949dbafafef8318194ef0" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39563 } } peers { permanent_uuid: "d03178c76c8949b0b2c53dcde35cde7c" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33607 } } peers { permanent_uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 38425 } }
I20260812 14:26:34.479192 27014 ts_tablet_manager.cc:1428] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260812 14:26:34.479249 26674 heartbeater.cc:503] Master 127.5.148.62:35611 was elected leader, sending a full tablet report...
I20260812 14:26:34.479729 27013 raft_consensus.cc:348] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efdae7898d949dbafafef8318194ef0" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39563 } } peers { permanent_uuid: "d03178c76c8949b0b2c53dcde35cde7c" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33607 } } peers { permanent_uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 38425 } }
I20260812 14:26:34.479902 27013 raft_consensus.cc:374] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:34.479949 27013 raft_consensus.cc:729] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efdae7898d949dbafafef8318194ef0, State: Initialized, Role: FOLLOWER
I20260812 14:26:34.480062 27013 consensus_queue.cc:260] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efdae7898d949dbafafef8318194ef0" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39563 } } peers { permanent_uuid: "d03178c76c8949b0b2c53dcde35cde7c" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33607 } } peers { permanent_uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 38425 } }
I20260812 14:26:34.480367 27013 ts_tablet_manager.cc:1428] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 14:26:34.480453 26992 heartbeater.cc:503] Master 127.5.148.62:35611 was elected leader, sending a full tablet report...
I20260812 14:26:34.524606 27021 raft_consensus.cc:482] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:26:34.524761 27021 raft_consensus.cc:504] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efdae7898d949dbafafef8318194ef0" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39563 } } peers { permanent_uuid: "d03178c76c8949b0b2c53dcde35cde7c" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33607 } } peers { permanent_uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 38425 } }
I20260812 14:26:34.525174 27021 leader_election.cc:288] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efdae7898d949dbafafef8318194ef0 (127.5.148.3:39563), d03178c76c8949b0b2c53dcde35cde7c (127.5.148.2:33607)
I20260812 14:26:34.529277 27011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.525591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:36741 (local address 127.5.148.2:33607)
0812 14:26:34.525644 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:34.525646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:34.525752 (+ 106us) server_negotiation.cc:407] Connection header received
0812 14:26:34.525806 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:34.525807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:34.525818 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:34.525883 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:34.526029 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.526704 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.528328 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.528915 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.528980 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.528991 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:34.529162 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 14:26:34.529204 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:26:34.529277 27007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.525478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:57723 (local address 127.5.148.3:39563)
0812 14:26:34.525529 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:34.525531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:34.525542 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:26:34.525615 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:34.525616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:34.525627 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:34.525695 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:34.525873 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.526677 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.528328 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.528896 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.528980 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.528991 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:34.529162 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 14:26:34.529204 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:26:34.529285 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.525383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:39563 (local address 127.5.148.1:57723)
0812 14:26:34.525454 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:34.525472 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:34.525541 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:34.525716 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:34.525717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:34.525719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:34.525801 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.525803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.526780 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.526780 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.528231 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.528235 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.528995 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.528996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.529075 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.529081 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 14:26:34.529141 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 14:26:34.529190 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":119}
I20260812 14:26:34.529292 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.525506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:33607 (local address 127.5.148.1:36741)
0812 14:26:34.525666 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:34.525678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:34.525746 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:34.525899 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:34.525900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:34.525902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:34.525960 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.525961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.526780 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.526781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.528231 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:34.528235 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.528996 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.528996 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:34.529086 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.529089 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:26:34.529141 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 14:26:34.529190 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":82,"thread_start_us":105,"threads_started":1}
I20260812 14:26:34.529726 26934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23c3520a56f4dafb4be78690e3f396d" candidate_uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efdae7898d949dbafafef8318194ef0" is_pre_election: true
I20260812 14:26:34.529922 26934 raft_consensus.cc:2454] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba101b7fbdc0460db14a55dfb5e4fdd7 in term 0.
I20260812 14:26:34.529975 26775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23c3520a56f4dafb4be78690e3f396d" candidate_uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03178c76c8949b0b2c53dcde35cde7c" is_pre_election: true
I20260812 14:26:34.530154 26775 raft_consensus.cc:2454] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba101b7fbdc0460db14a55dfb5e4fdd7 in term 0.
I20260812 14:26:34.530189 26525 leader_election.cc:302] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9efdae7898d949dbafafef8318194ef0, ba101b7fbdc0460db14a55dfb5e4fdd7; no voters:
I20260812 14:26:34.530304 27021 raft_consensus.cc:2781] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:26:34.530368 27021 raft_consensus.cc:482] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:26:34.530395 27021 raft_consensus.cc:3037] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:34.531126 27021 raft_consensus.cc:504] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efdae7898d949dbafafef8318194ef0" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39563 } } peers { permanent_uuid: "d03178c76c8949b0b2c53dcde35cde7c" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33607 } } peers { permanent_uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 38425 } }
I20260812 14:26:34.531267 27021 leader_election.cc:288] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [CANDIDATE]: Term 1 election: Requested vote from peers 9efdae7898d949dbafafef8318194ef0 (127.5.148.3:39563), d03178c76c8949b0b2c53dcde35cde7c (127.5.148.2:33607)
I20260812 14:26:34.531461 26775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23c3520a56f4dafb4be78690e3f396d" candidate_uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03178c76c8949b0b2c53dcde35cde7c"
I20260812 14:26:34.531494 26934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23c3520a56f4dafb4be78690e3f396d" candidate_uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efdae7898d949dbafafef8318194ef0"
I20260812 14:26:34.531541 26775 raft_consensus.cc:3037] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:34.531594 26934 raft_consensus.cc:3037] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:34.532382 26775 raft_consensus.cc:2454] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba101b7fbdc0460db14a55dfb5e4fdd7 in term 1.
I20260812 14:26:34.532563 26934 raft_consensus.cc:2454] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba101b7fbdc0460db14a55dfb5e4fdd7 in term 1.
I20260812 14:26:34.532558 26525 leader_election.cc:302] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba101b7fbdc0460db14a55dfb5e4fdd7, d03178c76c8949b0b2c53dcde35cde7c; no voters:
I20260812 14:26:34.532680 27021 raft_consensus.cc:2781] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:34.532855 27021 raft_consensus.cc:686] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [term 1 LEADER]: Becoming Leader. State: Replica: ba101b7fbdc0460db14a55dfb5e4fdd7, State: Running, Role: LEADER
I20260812 14:26:34.532972 27021 consensus_queue.cc:237] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efdae7898d949dbafafef8318194ef0" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39563 } } peers { permanent_uuid: "d03178c76c8949b0b2c53dcde35cde7c" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33607 } } peers { permanent_uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 38425 } }
I20260812 14:26:34.533857 26431 catalog_manager.cc:5495] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba101b7fbdc0460db14a55dfb5e4fdd7 (127.5.148.1). New cstate: current_term: 1 leader_uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efdae7898d949dbafafef8318194ef0" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d03178c76c8949b0b2c53dcde35cde7c" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba101b7fbdc0460db14a55dfb5e4fdd7" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 38425 } health_report { overall_health: HEALTHY } } }
I20260812 14:26:34.540769 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.538371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.5.148.1:38425)
0812 14:26:34.538426 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:34.538428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:34.538451 (+ 23us) server_negotiation.cc:407] Connection header received
0812 14:26:34.538487 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:34.538487 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:34.538504 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:34.538561 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:26:34.538660 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.539117 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.540005 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:34.540244 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:34.540293 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:34.540439 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:26:34.540535 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:26:34.540597 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:34.540653 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 14:26:34.540670 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:26:34.542632 26775 raft_consensus.cc:1261] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c [term 1 FOLLOWER]: Refusing update from remote peer ba101b7fbdc0460db14a55dfb5e4fdd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:26:34.542641 26934 raft_consensus.cc:1261] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0 [term 1 FOLLOWER]: Refusing update from remote peer ba101b7fbdc0460db14a55dfb5e4fdd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:26:34.543025 27024 consensus_queue.cc:1035] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efdae7898d949dbafafef8318194ef0" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 39563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:26:34.543167 27021 consensus_queue.cc:1035] T b23c3520a56f4dafb4be78690e3f396d P ba101b7fbdc0460db14a55dfb5e4fdd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03178c76c8949b0b2c53dcde35cde7c" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:26:34.546535 27029 mvcc.cc:205] Tried to move back new op lower bound from 7317687478443917312 to 7317687478407757824. Current Snapshot: MvccSnapshot[applied={T|T < 7317687478443917312 or (T in {7317687478443917312})}]
I20260812 14:26:34.548285 26430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:26:34.548398 26430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:26:34.549198 26430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d25a79b208d4d73bdff982dd3c478fc: Sending DeleteTablet for 3 replicas of tablet b23c3520a56f4dafb4be78690e3f396d
I20260812 14:26:34.549540 26586 tablet_service.cc:1514] Processing DeleteTablet for tablet b23c3520a56f4dafb4be78690e3f396d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:34 UTC) from {username='slave'} at 127.0.0.1:53434
I20260812 14:26:34.549540 26911 tablet_service.cc:1514] Processing DeleteTablet for tablet b23c3520a56f4dafb4be78690e3f396d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:34 UTC) from {username='slave'} at 127.0.0.1:60398
I20260812 14:26:34.549814 27042 tablet_replica.cc:330] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0: stopping tablet replica
I20260812 14:26:34.549787 26751 tablet_service.cc:1514] Processing DeleteTablet for tablet b23c3520a56f4dafb4be78690e3f396d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:34 UTC) from {username='slave'} at 127.0.0.1:53982
I20260812 14:26:34.549927 27042 raft_consensus.cc:2229] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:26:34.550041 27042 raft_consensus.cc:2258] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:26:34.550223 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 26508
I20260812 14:26:34.551206 27042 ts_tablet_manager.cc:1905] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:26:34.553436 27042 ts_tablet_manager.cc:1918] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:26:34.553524 27042 log.cc:1196] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/b23c3520a56f4dafb4be78690e3f396d
I20260812 14:26:34.553762 27042 ts_tablet_manager.cc:1939] T b23c3520a56f4dafb4be78690e3f396d P 9efdae7898d949dbafafef8318194ef0: Deleting consensus metadata
I20260812 14:26:34.553936 27043 tablet_replica.cc:330] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c: stopping tablet replica
I20260812 14:26:34.554077 27043 raft_consensus.cc:2229] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:26:34.554180 27043 raft_consensus.cc:2258] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:26:34.554418 26412 catalog_manager.cc:4841] TS 9efdae7898d949dbafafef8318194ef0 (127.5.148.3:39563): tablet b23c3520a56f4dafb4be78690e3f396d (table test-table [id=d75df6b6b28b48c38df5b004f96a575e]) successfully deleted
I20260812 14:26:34.554720 27043 ts_tablet_manager.cc:1905] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:26:34.556413 27043 ts_tablet_manager.cc:1918] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:26:34.556476 27043 log.cc:1196] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/b23c3520a56f4dafb4be78690e3f396d
I20260812 14:26:34.556743 27043 ts_tablet_manager.cc:1939] T b23c3520a56f4dafb4be78690e3f396d P d03178c76c8949b0b2c53dcde35cde7c: Deleting consensus metadata
I20260812 14:26:34.557215 26412 catalog_manager.cc:4841] TS d03178c76c8949b0b2c53dcde35cde7c (127.5.148.2:33607): tablet b23c3520a56f4dafb4be78690e3f396d (table test-table [id=d75df6b6b28b48c38df5b004f96a575e]) successfully deleted
W20260812 14:26:34.559680 26416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:38425 (error 108)
I20260812 14:26:34.560237 27004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:34.560076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:38425 (local address 127.0.0.1:53458)
0812 14:26:34.560159 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:34.560208 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:38425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 14:26:34.560240 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 26678
W20260812 14:26:34.560355 26416 catalog_manager.cc:4584] TS ba101b7fbdc0460db14a55dfb5e4fdd7 (127.5.148.1:38425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b23c3520a56f4dafb4be78690e3f396d: Network error: Client connection negotiation failed: client connection to 127.5.148.1:38425: connect: Connection refused (error 111)
I20260812 14:26:34.569209 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 26831
I20260812 14:26:34.578193 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 26397
2026-08-12T14:26:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (740 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:34 dist-test-slave-8w9v krb5kdc[27050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:34 dist-test-slave-8w9v krb5kdc[27050](info): set up 2 sockets
Aug 12 14:26:34 dist-test-slave-8w9v krb5kdc[27050](info): commencing operation
krb5kdc: starting...
W20260812 14:26:35.708252 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:35 dist-test-slave-8w9v krb5kdc[27050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:26:35.838650 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:32875
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:39395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:32875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:35.921504 27070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:35.921772 27070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:35.921814 27070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:35.923012 27070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:35.923091 27070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:35.923116 27070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:35.923138 27070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:35.923168 27070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:35.924703 27070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:39395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:32875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:32875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:35.925000 27070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:35.925246 27070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:35.930747 27082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:35.930716 27077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:35.930734 27078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:35.930855 27070 server_base.cc:1034] running on GCE node
I20260812 14:26:35.931289 27070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:35.931595 27070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:35.932782 27070 hybrid_clock.cc:648] HybridClock initialized: now 1786544795932753 us; error 45 us; skew 500 ppm
Aug 12 14:26:35 dist-test-slave-8w9v krb5kdc[27050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544795, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:35.935408 27070 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:26:35.935904 27070 webserver.cc:460] Webserver started at http://127.5.148.62:36333/ using document root <none> and password file <none>
I20260812 14:26:35.936226 27070 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:35.936276 27070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:35.936441 27070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:35.937546 27070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "6c99ae0712bc4e9987be58bf8e412b96"
format_stamp: "Formatted at 2026-08-12 14:26:35 on dist-test-slave-8w9v"
server_key: "dcaf1d4ca8e9b352fdfbd79b3a3ab6f2"
server_key_iv: "6a5a59497729b9686df48ec02211439f"
server_key_version: "encryptionkey@0"
I20260812 14:26:35.937924 27070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "6c99ae0712bc4e9987be58bf8e412b96"
format_stamp: "Formatted at 2026-08-12 14:26:35 on dist-test-slave-8w9v"
server_key: "dcaf1d4ca8e9b352fdfbd79b3a3ab6f2"
server_key_iv: "6a5a59497729b9686df48ec02211439f"
server_key_version: "encryptionkey@0"
I20260812 14:26:35.939304 27070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:26:35.940053 27094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:35.940259 27070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:26:35.940371 27070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "6c99ae0712bc4e9987be58bf8e412b96"
format_stamp: "Formatted at 2026-08-12 14:26:35 on dist-test-slave-8w9v"
server_key: "dcaf1d4ca8e9b352fdfbd79b3a3ab6f2"
server_key_iv: "6a5a59497729b9686df48ec02211439f"
server_key_version: "encryptionkey@0"
I20260812 14:26:35.940476 27070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:35.957506 27070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:35.959497 27070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:35.959688 27070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:35.964284 27070 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:32875
I20260812 14:26:35.964341 27161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:32875 every 8 connection(s)
I20260812 14:26:35.964740 27070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:35.965689 27162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:35.968163 27162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96: Bootstrap starting.
I20260812 14:26:35.968811 27162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:35.969117 27162 log.cc:824] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:35.969861 27162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96: No bootstrap required, opened a new log
I20260812 14:26:35.971503 27162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c99ae0712bc4e9987be58bf8e412b96" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32875 } }
I20260812 14:26:35.971669 27162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:35.971722 27162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c99ae0712bc4e9987be58bf8e412b96, State: Initialized, Role: FOLLOWER
I20260812 14:26:35.971915 27162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c99ae0712bc4e9987be58bf8e412b96" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32875 } }
I20260812 14:26:35.972026 27162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:35.972074 27162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:35.972149 27162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:35.972699 27162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c99ae0712bc4e9987be58bf8e412b96" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32875 } }
I20260812 14:26:35.972826 27162 leader_election.cc:302] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c99ae0712bc4e9987be58bf8e412b96; no voters:
I20260812 14:26:35.973012 27162 leader_election.cc:288] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:35.973076 27165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:35.973285 27165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [term 1 LEADER]: Becoming Leader. State: Replica: 6c99ae0712bc4e9987be58bf8e412b96, State: Running, Role: LEADER
I20260812 14:26:35.973284 27162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:35.973397 27165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c99ae0712bc4e9987be58bf8e412b96" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32875 } }
I20260812 14:26:35.973811 27166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c99ae0712bc4e9987be58bf8e412b96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c99ae0712bc4e9987be58bf8e412b96" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32875 } } }
I20260812 14:26:35.973919 27166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:35.974061 27168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c99ae0712bc4e9987be58bf8e412b96. Latest consensus state: current_term: 1 leader_uuid: "6c99ae0712bc4e9987be58bf8e412b96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c99ae0712bc4e9987be58bf8e412b96" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 32875 } } }
I20260812 14:26:35.974155 27176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:35.974184 27168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:35.974628 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 27070
I20260812 14:26:35.974771 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:35.974951 5712 external_mini_cluster.cc:1428] Setting key f685376682c39978d7d1fdb110109cd8
I20260812 14:26:35.975450 27176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:35.977324 27176 catalog_manager.cc:1347] Generated new cluster ID: 9d0de66e4a334c03a7fec77cb62f3d1c
I20260812 14:26:35.977376 27176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:26:35 dist-test-slave-8w9v krb5kdc[27050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:26:35.987654 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:35.975628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.5.148.62:32875)
0812 14:26:35.975803 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:35.975812 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:35.975848 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:26:35.975969 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:35.975977 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:35.976047 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:35.976217 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:35.977035 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:35.978411 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:35.979545 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:35.980217 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:35.982919 (+ 2702us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:35.983295 (+ 376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:35.983298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:35.983299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:35.983309 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:35.985954 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:35.986368 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:35.986370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:35.986371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:35.986493 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:35.986718 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:35.986718 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:35.986719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:35.987030 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:35.987160 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:35.987302 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 14:26:35.987434 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260812 14:26:35.992902 27176 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:35.993530 27176 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:35.999760 27176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96: Generated new TSK 0
I20260812 14:26:35.999945 27176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:26:36.035832 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:32875
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:39395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:36.123286 27193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:36.123512 27193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:36.123536 27193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:36.124766 27193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:36.124812 27193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:36.124873 27193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:26:36.126389 27193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:39395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:36.126683 27193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:36.126930 27193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:36.132071 27198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:36.132071 27202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:36.132175 27200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:36.132412 27193 server_base.cc:1034] running on GCE node
I20260812 14:26:36.132601 27193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:36.132792 27193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:36.133935 27193 hybrid_clock.cc:648] HybridClock initialized: now 1786544796133923 us; error 35 us; skew 500 ppm
Aug 12 14:26:36 dist-test-slave-8w9v krb5kdc[27050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544796, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:36.136618 27193 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:26:36.137113 27193 webserver.cc:460] Webserver started at http://127.5.148.1:35941/ using document root <none> and password file <none>
I20260812 14:26:36.137368 27193 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:36.137414 27193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:36.137615 27193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:36.138696 27193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "4faf9a895d724871a73ecfed4e6e22d2"
format_stamp: "Formatted at 2026-08-12 14:26:36 on dist-test-slave-8w9v"
server_key: "e070643d46c6646e39e5bb34e9685c64"
server_key_iv: "4fc255f7a0b4b0e57fd49099b6896ee7"
server_key_version: "encryptionkey@0"
I20260812 14:26:36.139061 27193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "4faf9a895d724871a73ecfed4e6e22d2"
format_stamp: "Formatted at 2026-08-12 14:26:36 on dist-test-slave-8w9v"
server_key: "e070643d46c6646e39e5bb34e9685c64"
server_key_iv: "4fc255f7a0b4b0e57fd49099b6896ee7"
server_key_version: "encryptionkey@0"
I20260812 14:26:36.140501 27193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:26:36.141224 27212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:36.141425 27193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:26:36.141562 27193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "4faf9a895d724871a73ecfed4e6e22d2"
format_stamp: "Formatted at 2026-08-12 14:26:36 on dist-test-slave-8w9v"
server_key: "e070643d46c6646e39e5bb34e9685c64"
server_key_iv: "4fc255f7a0b4b0e57fd49099b6896ee7"
server_key_version: "encryptionkey@0"
I20260812 14:26:36.141638 27193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:36.153231 27193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:36.154999 27193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:36.155182 27193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:36.155419 27193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:36.155730 27193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:36.155784 27193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:36.155829 27193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:36.155854 27193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:36.162117 27193 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:43751
I20260812 14:26:36.162170 27359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:43751 every 8 connection(s)
I20260812 14:26:36.162513 27193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:26:36 dist-test-slave-8w9v krb5kdc[27050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544796, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:26:36.171468 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.163297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:45287 (local address 127.5.148.62:32875)
0812 14:26:36.163365 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:36.163367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:36.163574 (+ 207us) server_negotiation.cc:407] Connection header received
0812 14:26:36.164076 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:36.164077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:36.164090 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:36.164175 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:36.165141 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.165631 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.166617 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.166819 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.166874 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.169031 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:36.169032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:36.169032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:36.169039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:36.170325 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:36.170681 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:36.170682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:36.170682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:36.170731 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:36.170947 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:36.170948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:36.170948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:36.171048 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:36.171098 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:36.171322 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 14:26:36.171401 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 14:26:36.171638 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.163286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:32875 (local address 127.5.148.1:45287)
0812 14:26:36.163462 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:36.163501 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:36.163983 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:36.164266 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:36.164270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:36.164850 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:36.165055 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.165060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.165710 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.165710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.166532 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.166534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.166894 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.166895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.167004 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.167487 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:26:36.167495 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:36.168942 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:36.170397 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:36.170404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:36.170406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:36.170616 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:36.170787 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:36.170788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:36.170788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:36.170886 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:36.171100 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:36.171105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:36.171216 (+ 111us) client_negotiation.cc:769] Sending connection context
0812 14:26:36.171295 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 14:26:36.171414 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260812 14:26:36.172288 27360 heartbeater.cc:347] Connected to a master server at 127.5.148.62:32875
I20260812 14:26:36.172413 27360 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:36.172616 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 27193
I20260812 14:26:36.172653 27360 heartbeater.cc:511] Master 127.5.148.62:32875 requested a full tablet report, sending...
I20260812 14:26:36.172729 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:26:36.172951 5712 external_mini_cluster.cc:1428] Setting key ca5a4e176cec4e4413cf911ec342764e
I20260812 14:26:36.173262 27118 ts_manager.cc:194] Registered new tserver with Master: 4faf9a895d724871a73ecfed4e6e22d2 (127.5.148.1:43751)
I20260812 14:26:36.174145 27118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:45287
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:26:36.199852 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:32875
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:39395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:36.285741 27366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:36.285956 27366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:36.285981 27366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:36.287079 27366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:36.287115 27366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:36.287140 27366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:26:36.288556 27366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:39395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:36.288738 27366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:36.288916 27366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:36.294106 27372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:36.294157 27371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:36.294122 27366 server_base.cc:1034] running on GCE node
W20260812 14:26:36.294106 27375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:36.294521 27366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:36.294782 27366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:36.295956 27366 hybrid_clock.cc:648] HybridClock initialized: now 1786544796295941 us; error 52 us; skew 500 ppm
Aug 12 14:26:36 dist-test-slave-8w9v krb5kdc[27050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544796, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:36.298503 27366 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:26:36.298954 27366 webserver.cc:460] Webserver started at http://127.5.148.2:41417/ using document root <none> and password file <none>
I20260812 14:26:36.299188 27366 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:36.299256 27366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:36.299422 27366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:36.300621 27366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "89ca27104d3143f1af262ee243f22fb3"
format_stamp: "Formatted at 2026-08-12 14:26:36 on dist-test-slave-8w9v"
server_key: "4a4e6bda630db54d00e2752f691f9fc5"
server_key_iv: "3d7d092c14c4fd431c01df4ec883146d"
server_key_version: "encryptionkey@0"
I20260812 14:26:36.301010 27366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "89ca27104d3143f1af262ee243f22fb3"
format_stamp: "Formatted at 2026-08-12 14:26:36 on dist-test-slave-8w9v"
server_key: "4a4e6bda630db54d00e2752f691f9fc5"
server_key_iv: "3d7d092c14c4fd431c01df4ec883146d"
server_key_version: "encryptionkey@0"
I20260812 14:26:36.302368 27366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:26:36.303118 27385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:36.303290 27366 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:26:36.303401 27366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "89ca27104d3143f1af262ee243f22fb3"
format_stamp: "Formatted at 2026-08-12 14:26:36 on dist-test-slave-8w9v"
server_key: "4a4e6bda630db54d00e2752f691f9fc5"
server_key_iv: "3d7d092c14c4fd431c01df4ec883146d"
server_key_version: "encryptionkey@0"
I20260812 14:26:36.303481 27366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:36.312891 27366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:36.315276 27366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:36.315451 27366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:36.315690 27366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:36.316094 27366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:36.316166 27366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:36.316210 27366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:36.316241 27366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:36.322304 27366 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:41331
I20260812 14:26:36.322355 27525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:41331 every 8 connection(s)
I20260812 14:26:36.322698 27366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:26:36.326247 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 27366
I20260812 14:26:36.326402 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:26:36.326608 5712 external_mini_cluster.cc:1428] Setting key 606441f049279f672ac85f054335b5ef
Aug 12 14:26:36 dist-test-slave-8w9v krb5kdc[27050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544796, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:26:36.333271 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.323441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:48097 (local address 127.5.148.62:32875)
0812 14:26:36.323504 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:36.323506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:36.323705 (+ 199us) server_negotiation.cc:407] Connection header received
0812 14:26:36.324342 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:36.324344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:36.324356 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:36.324408 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:36.325427 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.325961 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.327446 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.327637 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.327731 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.330003 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:36.330004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:36.330005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:36.330014 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:36.331617 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:36.332225 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:36.332226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:36.332226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:36.332280 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:36.332563 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:36.332564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:36.332564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:36.332692 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:36.332764 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:36.333075 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 14:26:36.333177 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:26:36.333470 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.323386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:32875 (local address 127.5.148.2:48097)
0812 14:26:36.323579 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:36.323625 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:36.324192 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:36.324515 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:36.324519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:36.325113 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:36.325313 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.325317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.326050 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.326051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.327257 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.327260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.327742 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.327750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.327895 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.328489 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:26:36.328498 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:36.329909 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:36.331706 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:36.331715 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:36.331717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:36.332148 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:36.332354 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:36.332355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:36.332356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:36.332483 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:36.332768 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:36.332776 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:36.332934 (+ 158us) client_negotiation.cc:769] Sending connection context
0812 14:26:36.333039 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 14:26:36.333224 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260812 14:26:36.334025 27526 heartbeater.cc:347] Connected to a master server at 127.5.148.62:32875
I20260812 14:26:36.334147 27526 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:36.334375 27526 heartbeater.cc:511] Master 127.5.148.62:32875 requested a full tablet report, sending...
I20260812 14:26:36.334798 27118 ts_manager.cc:194] Registered new tserver with Master: 89ca27104d3143f1af262ee243f22fb3 (127.5.148.2:41331)
I20260812 14:26:36.335337 27118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:48097
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:26:36.355135 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:32875
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:39395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:36.440716 27531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:36.440933 27531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:36.440959 27531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:36.442179 27531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:36.442226 27531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:36.442286 27531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:26:36.444039 27531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:39395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:36.444331 27531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:36.444624 27531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:36.450101 27538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:36.450109 27542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:36.450116 27539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:36.450620 27531 server_base.cc:1034] running on GCE node
I20260812 14:26:36.450805 27531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:36.451009 27531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:36.452183 27531 hybrid_clock.cc:648] HybridClock initialized: now 1786544796452166 us; error 29 us; skew 500 ppm
Aug 12 14:26:36 dist-test-slave-8w9v krb5kdc[27050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544796, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:36.454815 27531 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:26:36.455242 27531 webserver.cc:460] Webserver started at http://127.5.148.3:35341/ using document root <none> and password file <none>
I20260812 14:26:36.455461 27531 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:36.455503 27531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:36.455595 27531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:36.456705 27531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "868e7aebc5814b78a659d21b0cf72e35"
format_stamp: "Formatted at 2026-08-12 14:26:36 on dist-test-slave-8w9v"
server_key: "2690a7326e1c457cec9b0bd89bbbd4e6"
server_key_iv: "99bb9d91141fad84db3561832ce436c6"
server_key_version: "encryptionkey@0"
I20260812 14:26:36.457010 27531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "868e7aebc5814b78a659d21b0cf72e35"
format_stamp: "Formatted at 2026-08-12 14:26:36 on dist-test-slave-8w9v"
server_key: "2690a7326e1c457cec9b0bd89bbbd4e6"
server_key_iv: "99bb9d91141fad84db3561832ce436c6"
server_key_version: "encryptionkey@0"
I20260812 14:26:36.458308 27531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:26:36.458999 27549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:36.459172 27531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:26:36.459275 27531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "868e7aebc5814b78a659d21b0cf72e35"
format_stamp: "Formatted at 2026-08-12 14:26:36 on dist-test-slave-8w9v"
server_key: "2690a7326e1c457cec9b0bd89bbbd4e6"
server_key_iv: "99bb9d91141fad84db3561832ce436c6"
server_key_version: "encryptionkey@0"
I20260812 14:26:36.459350 27531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:36.480332 27531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:36.482915 27531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:36.483091 27531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:36.483351 27531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:36.483685 27531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:36.483737 27531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:36.483780 27531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:36.483812 27531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:36.490430 27531 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:32891
I20260812 14:26:36.490475 27686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:32891 every 8 connection(s)
I20260812 14:26:36.490842 27531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:26:36.491295 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 27531
I20260812 14:26:36.491425 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:26:36.491655 5712 external_mini_cluster.cc:1428] Setting key 0cba8d1844366f56c6b121f2b191fecc
Aug 12 14:26:36 dist-test-slave-8w9v krb5kdc[27050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544796, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:26:36.500752 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.491704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:42343 (local address 127.5.148.62:32875)
0812 14:26:36.491768 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:36.491770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:36.492149 (+ 379us) server_negotiation.cc:407] Connection header received
0812 14:26:36.492862 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:36.492862 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:36.492872 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:36.492921 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:36.493852 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.494368 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.495369 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.495562 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.495615 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.497850 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:36.497851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:36.497852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:36.497859 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:36.499336 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:36.499757 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:36.499758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:36.499759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:36.499836 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:36.500115 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:36.500116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:36.500116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:36.500276 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:36.500329 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:36.500613 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 14:26:36.500680 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:26:36.500972 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.491668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:32875 (local address 127.5.148.3:42343)
0812 14:26:36.492005 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:36.492060 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:36.492771 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:36.492978 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:36.492982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:36.493573 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:36.493772 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.493777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.494460 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.494461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.495296 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.495299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.495633 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.495634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.495759 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.496356 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:26:36.496365 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:36.497756 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:36.499419 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:36.499426 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:36.499428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:36.499661 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:36.499922 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:36.499923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:36.499924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:36.500054 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:36.500333 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:36.500338 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:36.500489 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 14:26:36.500580 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 14:26:36.500715 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":100,"threads_started":1}
I20260812 14:26:36.501505 27687 heartbeater.cc:347] Connected to a master server at 127.5.148.62:32875
I20260812 14:26:36.501626 27687 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:36.501825 27687 heartbeater.cc:511] Master 127.5.148.62:32875 requested a full tablet report, sending...
I20260812 14:26:36.502254 27118 ts_manager.cc:194] Registered new tserver with Master: 868e7aebc5814b78a659d21b0cf72e35 (127.5.148.3:32891)
I20260812 14:26:36.502653 27118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:42343
I20260812 14:26:36.503460 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:26:36.511667 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.504926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60400 (local address 127.5.148.62:32875)
0812 14:26:36.504991 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:36.504993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:36.505122 (+ 129us) server_negotiation.cc:407] Connection header received
0812 14:26:36.505197 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:36.505199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:36.505209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:36.505264 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:36.505845 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.506400 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.507144 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.507365 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.507450 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.508960 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:36.508961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:36.508962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:36.508969 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:36.510347 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:36.510792 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:36.510794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:36.510794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:36.510855 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:36.511095 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:36.511096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:36.511097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:36.511251 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:36.511373 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:36.511488 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 14:26:36.511556 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 14:26:36.513427 27118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:26:36.515056 27118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:26:36.522014 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.517576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54470 (local address 127.5.148.2:41331)
0812 14:26:36.517938 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:36.517943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:36.517953 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:26:36.517996 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:36.517998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:36.518010 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:36.518138 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:36.518351 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.519540 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.520800 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.521544 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.521625 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.521708 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:36.521875 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 14:26:36.521922 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":81,"threads_started":1}
I20260812 14:26:36.522018 27707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.517665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:43751 (local address 127.0.0.1:49882)
0812 14:26:36.517852 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:36.517862 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:36.517911 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:36.518517 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:36.518518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:36.518520 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:36.518568 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.518570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.519448 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.519449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.520869 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.520873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.521768 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.521768 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.521831 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.521835 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:26:36.521892 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 14:26:36.521939 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":37,"thread_start_us":76,"threads_started":1}
I20260812 14:26:36.522018 27708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.517424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:41331 (local address 127.0.0.1:54470)
0812 14:26:36.517639 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:36.517681 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:36.517754 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:36.518149 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:36.518153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:36.518156 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:36.518251 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.518254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.519683 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.519684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.520687 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.520690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.521633 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.521634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.521742 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.521759 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 14:26:36.521837 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 14:26:36.521909 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260812 14:26:36.522513 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.517935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49882 (local address 127.5.148.1:43751)
0812 14:26:36.518337 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:36.518341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:36.518352 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:26:36.518394 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:36.518396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:36.518411 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:36.518515 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:36.518630 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.519333 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.520964 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.521674 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.521751 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.521809 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:36.522377 (+ 568us) server_negotiation.cc:299] Negotiation successful
0812 14:26:36.522433 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":109,"threads_started":1}
I20260812 14:26:36.522958 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.517445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:32891 (local address 127.0.0.1:42534)
0812 14:26:36.517789 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:36.517801 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:36.517855 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:36.518223 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:36.518224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:36.518226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:36.518265 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.518266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.519646 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.519647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.520869 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.520873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.522717 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.522718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.522784 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.522789 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:26:36.522822 (+ 33us) client_negotiation.cc:240] Negotiation successful
0812 14:26:36.522868 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":45,"thread_start_us":109,"threads_started":1}
I20260812 14:26:36.522979 27448 tablet_service.cc:1467] Processing CreateTablet for tablet 11d501a5bb834e708d51370af5cd0436 (DEFAULT_TABLE table=test-table [id=8583107ed63a4cbeabde7c1d6ed84ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:36.523386 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.517533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42534 (local address 127.5.148.3:32891)
0812 14:26:36.518043 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:36.518048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:36.518059 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:26:36.518101 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:36.518103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:36.518120 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:36.518344 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:36.518427 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.519539 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.521875 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.522621 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.523137 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.523217 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:36.523244 (+ 27us) server_negotiation.cc:299] Negotiation successful
0812 14:26:36.523295 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":90,"threads_started":1}
I20260812 14:26:36.523406 27448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d501a5bb834e708d51370af5cd0436. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:36.523358 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 11d501a5bb834e708d51370af5cd0436 (DEFAULT_TABLE table=test-table [id=8583107ed63a4cbeabde7c1d6ed84ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:36.523765 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d501a5bb834e708d51370af5cd0436. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:36.524057 27607 tablet_service.cc:1467] Processing CreateTablet for tablet 11d501a5bb834e708d51370af5cd0436 (DEFAULT_TABLE table=test-table [id=8583107ed63a4cbeabde7c1d6ed84ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:36.524428 27607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d501a5bb834e708d51370af5cd0436. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:36.526860 27713 tablet_bootstrap.cc:492] T 11d501a5bb834e708d51370af5cd0436 P 4faf9a895d724871a73ecfed4e6e22d2: Bootstrap starting.
I20260812 14:26:36.527070 27714 tablet_bootstrap.cc:492] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3: Bootstrap starting.
I20260812 14:26:36.527845 27715 tablet_bootstrap.cc:492] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35: Bootstrap starting.
I20260812 14:26:36.527884 27714 tablet_bootstrap.cc:654] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:36.527884 27713 tablet_bootstrap.cc:654] T 11d501a5bb834e708d51370af5cd0436 P 4faf9a895d724871a73ecfed4e6e22d2: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:36.528530 27715 tablet_bootstrap.cc:654] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:36.528644 27714 log.cc:824] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:36.528649 27713 log.cc:824] T 11d501a5bb834e708d51370af5cd0436 P 4faf9a895d724871a73ecfed4e6e22d2: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:36.529187 27715 log.cc:824] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:36.529480 27714 tablet_bootstrap.cc:492] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3: No bootstrap required, opened a new log
I20260812 14:26:36.529480 27713 tablet_bootstrap.cc:492] T 11d501a5bb834e708d51370af5cd0436 P 4faf9a895d724871a73ecfed4e6e22d2: No bootstrap required, opened a new log
I20260812 14:26:36.529558 27714 ts_tablet_manager.cc:1397] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 14:26:36.529558 27713 ts_tablet_manager.cc:1397] T 11d501a5bb834e708d51370af5cd0436 P 4faf9a895d724871a73ecfed4e6e22d2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260812 14:26:36.529991 27715 tablet_bootstrap.cc:492] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35: No bootstrap required, opened a new log
I20260812 14:26:36.530131 27715 ts_tablet_manager.cc:1397] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:26:36.531591 27715 raft_consensus.cc:348] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ca27104d3143f1af262ee243f22fb3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41331 } } peers { permanent_uuid: "868e7aebc5814b78a659d21b0cf72e35" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 32891 } } peers { permanent_uuid: "4faf9a895d724871a73ecfed4e6e22d2" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43751 } }
I20260812 14:26:36.531765 27715 raft_consensus.cc:374] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:36.531651 27713 raft_consensus.cc:348] T 11d501a5bb834e708d51370af5cd0436 P 4faf9a895d724871a73ecfed4e6e22d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ca27104d3143f1af262ee243f22fb3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41331 } } peers { permanent_uuid: "868e7aebc5814b78a659d21b0cf72e35" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 32891 } } peers { permanent_uuid: "4faf9a895d724871a73ecfed4e6e22d2" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43751 } }
I20260812 14:26:36.531651 27714 raft_consensus.cc:348] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ca27104d3143f1af262ee243f22fb3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41331 } } peers { permanent_uuid: "868e7aebc5814b78a659d21b0cf72e35" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 32891 } } peers { permanent_uuid: "4faf9a895d724871a73ecfed4e6e22d2" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43751 } }
I20260812 14:26:36.531813 27715 raft_consensus.cc:729] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868e7aebc5814b78a659d21b0cf72e35, State: Initialized, Role: FOLLOWER
I20260812 14:26:36.531832 27713 raft_consensus.cc:374] T 11d501a5bb834e708d51370af5cd0436 P 4faf9a895d724871a73ecfed4e6e22d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:36.531832 27714 raft_consensus.cc:374] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:36.531869 27714 raft_consensus.cc:729] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ca27104d3143f1af262ee243f22fb3, State: Initialized, Role: FOLLOWER
I20260812 14:26:36.531869 27713 raft_consensus.cc:729] T 11d501a5bb834e708d51370af5cd0436 P 4faf9a895d724871a73ecfed4e6e22d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4faf9a895d724871a73ecfed4e6e22d2, State: Initialized, Role: FOLLOWER
I20260812 14:26:36.531975 27715 consensus_queue.cc:260] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ca27104d3143f1af262ee243f22fb3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41331 } } peers { permanent_uuid: "868e7aebc5814b78a659d21b0cf72e35" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 32891 } } peers { permanent_uuid: "4faf9a895d724871a73ecfed4e6e22d2" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43751 } }
I20260812 14:26:36.531996 27713 consensus_queue.cc:260] T 11d501a5bb834e708d51370af5cd0436 P 4faf9a895d724871a73ecfed4e6e22d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ca27104d3143f1af262ee243f22fb3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41331 } } peers { permanent_uuid: "868e7aebc5814b78a659d21b0cf72e35" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 32891 } } peers { permanent_uuid: "4faf9a895d724871a73ecfed4e6e22d2" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43751 } }
I20260812 14:26:36.532047 27714 consensus_queue.cc:260] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ca27104d3143f1af262ee243f22fb3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41331 } } peers { permanent_uuid: "868e7aebc5814b78a659d21b0cf72e35" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 32891 } } peers { permanent_uuid: "4faf9a895d724871a73ecfed4e6e22d2" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43751 } }
I20260812 14:26:36.532331 27713 ts_tablet_manager.cc:1428] T 11d501a5bb834e708d51370af5cd0436 P 4faf9a895d724871a73ecfed4e6e22d2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 14:26:36.532326 27715 ts_tablet_manager.cc:1428] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:26:36.532377 27526 heartbeater.cc:503] Master 127.5.148.62:32875 was elected leader, sending a full tablet report...
I20260812 14:26:36.532469 27687 heartbeater.cc:503] Master 127.5.148.62:32875 was elected leader, sending a full tablet report...
I20260812 14:26:36.532485 27360 heartbeater.cc:503] Master 127.5.148.62:32875 was elected leader, sending a full tablet report...
I20260812 14:26:36.532331 27714 ts_tablet_manager.cc:1428] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 14:26:36.547289 27720 raft_consensus.cc:482] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:26:36.547434 27720 raft_consensus.cc:504] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ca27104d3143f1af262ee243f22fb3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41331 } } peers { permanent_uuid: "868e7aebc5814b78a659d21b0cf72e35" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 32891 } } peers { permanent_uuid: "4faf9a895d724871a73ecfed4e6e22d2" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43751 } }
I20260812 14:26:36.547802 27720 leader_election.cc:288] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89ca27104d3143f1af262ee243f22fb3 (127.5.148.2:41331), 4faf9a895d724871a73ecfed4e6e22d2 (127.5.148.1:43751)
I20260812 14:26:36.551795 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.548072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:56931 (local address 127.5.148.2:41331)
0812 14:26:36.548147 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:36.548149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:36.548159 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:26:36.548194 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:36.548195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:36.548208 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:36.548275 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:36.548500 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.549272 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.550807 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.551409 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.551491 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.551503 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:36.551676 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 14:26:36.551717 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:26:36.551795 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.548095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:43869 (local address 127.5.148.1:43751)
0812 14:26:36.548172 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:36.548174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:36.548341 (+ 167us) server_negotiation.cc:407] Connection header received
0812 14:26:36.548410 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:36.548411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:36.548424 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:36.548495 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:36.548634 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.549321 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.550802 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.551409 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.551491 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.551503 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:36.551673 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 14:26:36.551716 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 14:26:36.551821 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.547968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:41331 (local address 127.5.148.3:56931)
0812 14:26:36.548048 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:36.548060 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:36.548104 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:36.548307 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:36.548308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:36.548310 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:36.548407 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.548409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.549377 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.549378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.550702 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.550706 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.551505 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.551506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.551591 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.551593 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:26:36.551652 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:26:36.551711 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":12}
I20260812 14:26:36.551831 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.548006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:43751 (local address 127.5.148.3:43869)
0812 14:26:36.548281 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:36.548291 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:36.548347 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:36.548509 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:36.548510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:36.548511 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:36.548568 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.548569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.549393 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.549393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.550702 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.550706 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.551505 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.551506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.551588 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.551593 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:26:36.551652 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:26:36.551711 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":105,"thread_start_us":165,"threads_started":1}
I20260812 14:26:36.552268 27473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d501a5bb834e708d51370af5cd0436" candidate_uuid: "868e7aebc5814b78a659d21b0cf72e35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89ca27104d3143f1af262ee243f22fb3" is_pre_election: true
I20260812 14:26:36.552497 27473 raft_consensus.cc:2454] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 868e7aebc5814b78a659d21b0cf72e35 in term 0.
I20260812 14:26:36.552481 27305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d501a5bb834e708d51370af5cd0436" candidate_uuid: "868e7aebc5814b78a659d21b0cf72e35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4faf9a895d724871a73ecfed4e6e22d2" is_pre_election: true
I20260812 14:26:36.552640 27305 raft_consensus.cc:2454] T 11d501a5bb834e708d51370af5cd0436 P 4faf9a895d724871a73ecfed4e6e22d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 868e7aebc5814b78a659d21b0cf72e35 in term 0.
I20260812 14:26:36.552793 27553 leader_election.cc:302] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868e7aebc5814b78a659d21b0cf72e35, 89ca27104d3143f1af262ee243f22fb3; no voters:
I20260812 14:26:36.553000 27720 raft_consensus.cc:2781] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:26:36.553043 27720 raft_consensus.cc:482] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:26:36.553061 27720 raft_consensus.cc:3037] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:36.553785 27720 raft_consensus.cc:504] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ca27104d3143f1af262ee243f22fb3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41331 } } peers { permanent_uuid: "868e7aebc5814b78a659d21b0cf72e35" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 32891 } } peers { permanent_uuid: "4faf9a895d724871a73ecfed4e6e22d2" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43751 } }
I20260812 14:26:36.553938 27720 leader_election.cc:288] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [CANDIDATE]: Term 1 election: Requested vote from peers 89ca27104d3143f1af262ee243f22fb3 (127.5.148.2:41331), 4faf9a895d724871a73ecfed4e6e22d2 (127.5.148.1:43751)
I20260812 14:26:36.554109 27305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d501a5bb834e708d51370af5cd0436" candidate_uuid: "868e7aebc5814b78a659d21b0cf72e35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4faf9a895d724871a73ecfed4e6e22d2"
I20260812 14:26:36.554126 27473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d501a5bb834e708d51370af5cd0436" candidate_uuid: "868e7aebc5814b78a659d21b0cf72e35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89ca27104d3143f1af262ee243f22fb3"
I20260812 14:26:36.554224 27473 raft_consensus.cc:3037] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:36.554225 27305 raft_consensus.cc:3037] T 11d501a5bb834e708d51370af5cd0436 P 4faf9a895d724871a73ecfed4e6e22d2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:36.555094 27305 raft_consensus.cc:2454] T 11d501a5bb834e708d51370af5cd0436 P 4faf9a895d724871a73ecfed4e6e22d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868e7aebc5814b78a659d21b0cf72e35 in term 1.
I20260812 14:26:36.555121 27473 raft_consensus.cc:2454] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868e7aebc5814b78a659d21b0cf72e35 in term 1.
I20260812 14:26:36.555298 27553 leader_election.cc:302] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868e7aebc5814b78a659d21b0cf72e35, 89ca27104d3143f1af262ee243f22fb3; no voters:
I20260812 14:26:36.555419 27720 raft_consensus.cc:2781] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:36.555589 27720 raft_consensus.cc:686] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [term 1 LEADER]: Becoming Leader. State: Replica: 868e7aebc5814b78a659d21b0cf72e35, State: Running, Role: LEADER
I20260812 14:26:36.555696 27720 consensus_queue.cc:237] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ca27104d3143f1af262ee243f22fb3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41331 } } peers { permanent_uuid: "868e7aebc5814b78a659d21b0cf72e35" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 32891 } } peers { permanent_uuid: "4faf9a895d724871a73ecfed4e6e22d2" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43751 } }
I20260812 14:26:36.556644 27117 catalog_manager.cc:5495] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 868e7aebc5814b78a659d21b0cf72e35 (127.5.148.3). New cstate: current_term: 1 leader_uuid: "868e7aebc5814b78a659d21b0cf72e35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ca27104d3143f1af262ee243f22fb3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "868e7aebc5814b78a659d21b0cf72e35" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 32891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4faf9a895d724871a73ecfed4e6e22d2" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43751 } health_report { overall_health: UNKNOWN } } }
I20260812 14:26:36.568981 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.566420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.5.148.3:32891)
0812 14:26:36.566485 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:36.566487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:36.566510 (+ 23us) server_negotiation.cc:407] Connection header received
0812 14:26:36.566592 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:36.566592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:36.566612 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:36.566688 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:26:36.566848 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.567327 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.568091 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.568338 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.568418 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.568573 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:26:36.568679 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:26:36.568790 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:36.568848 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 14:26:36.568857 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 14:26:36.570992 27473 raft_consensus.cc:1261] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3 [term 1 FOLLOWER]: Refusing update from remote peer 868e7aebc5814b78a659d21b0cf72e35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:26:36.571338 27305 raft_consensus.cc:1261] T 11d501a5bb834e708d51370af5cd0436 P 4faf9a895d724871a73ecfed4e6e22d2 [term 1 FOLLOWER]: Refusing update from remote peer 868e7aebc5814b78a659d21b0cf72e35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:26:36.571445 27725 consensus_queue.cc:1035] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89ca27104d3143f1af262ee243f22fb3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:26:36.571630 27725 consensus_queue.cc:1035] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4faf9a895d724871a73ecfed4e6e22d2" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 43751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 14:26:36.573235 27527 tablet.cc:2369] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:26:36.573967 27730 mvcc.cc:205] Tried to move back new op lower bound from 7317687486751649792 to 7317687486693011456. Current Snapshot: MvccSnapshot[applied={T|T < 7317687486751649792}]
I20260812 14:26:36.574664 27732 mvcc.cc:205] Tried to move back new op lower bound from 7317687486751649792 to 7317687486693011456. Current Snapshot: MvccSnapshot[applied={T|T < 7317687486751649792 or (T in {7317687486751649792})}]
I20260812 14:26:36.576930 27117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:26:36.577044 27117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:26:36.577843 27117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c99ae0712bc4e9987be58bf8e412b96: Sending DeleteTablet for 3 replicas of tablet 11d501a5bb834e708d51370af5cd0436
I20260812 14:26:36.578191 27448 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d501a5bb834e708d51370af5cd0436 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:36 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:54470
I20260812 14:26:36.578274 27607 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d501a5bb834e708d51370af5cd0436 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:36 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:42534
I20260812 14:26:36.578506 27274 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d501a5bb834e708d51370af5cd0436 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:36 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:49882
I20260812 14:26:36.578544 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 27193
I20260812 14:26:36.579207 27742 tablet_replica.cc:330] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35: stopping tablet replica
I20260812 14:26:36.579331 27742 raft_consensus.cc:2229] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:26:36.579483 27742 raft_consensus.cc:2258] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:26:36.580201 27741 tablet_replica.cc:330] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3: stopping tablet replica
I20260812 14:26:36.580327 27741 raft_consensus.cc:2229] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:26:36.580456 27741 raft_consensus.cc:2258] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:26:36.580646 27742 ts_tablet_manager.cc:1905] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:26:36.580940 27741 ts_tablet_manager.cc:1905] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:26:36.582597 27742 ts_tablet_manager.cc:1918] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:26:36.582720 27741 ts_tablet_manager.cc:1918] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:26:36.582794 27741 log.cc:1196] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/11d501a5bb834e708d51370af5cd0436
I20260812 14:26:36.582875 27742 log.cc:1196] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/11d501a5bb834e708d51370af5cd0436
I20260812 14:26:36.583109 27741 ts_tablet_manager.cc:1939] T 11d501a5bb834e708d51370af5cd0436 P 89ca27104d3143f1af262ee243f22fb3: Deleting consensus metadata
I20260812 14:26:36.583323 27742 ts_tablet_manager.cc:1939] T 11d501a5bb834e708d51370af5cd0436 P 868e7aebc5814b78a659d21b0cf72e35: Deleting consensus metadata
I20260812 14:26:36.583608 27097 catalog_manager.cc:4841] TS 89ca27104d3143f1af262ee243f22fb3 (127.5.148.2:41331): tablet 11d501a5bb834e708d51370af5cd0436 (table test-table [id=8583107ed63a4cbeabde7c1d6ed84ad1]) successfully deleted
I20260812 14:26:36.583762 27098 catalog_manager.cc:4841] TS 868e7aebc5814b78a659d21b0cf72e35 (127.5.148.3:32891): tablet 11d501a5bb834e708d51370af5cd0436 (table test-table [id=8583107ed63a4cbeabde7c1d6ed84ad1]) successfully deleted
W20260812 14:26:36.588760 27098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:43751 (error 108)
I20260812 14:26:36.589409 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.589251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:43751 (local address 127.0.0.1:49892)
0812 14:26:36.589316 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:36.589359 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 14:26:36.589493 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 27366
W20260812 14:26:36.589542 27098 catalog_manager.cc:4584] TS 4faf9a895d724871a73ecfed4e6e22d2 (127.5.148.1:43751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11d501a5bb834e708d51370af5cd0436: Network error: Client connection negotiation failed: client connection to 127.5.148.1:43751: connect: Connection refused (error 111)
I20260812 14:26:36.597335 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 27531
I20260812 14:26:36.606720 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 27070
2026-08-12T14:26:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2029 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T14:26:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:36Z Disabled control of system clock
I20260812 14:26:36.639707 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:41741
--webserver_interface=127.5.148.62
--webserver_port=0
--builtin_ntp_servers=127.5.148.20:40369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:41741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:26:36.723897 27747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:36.724152 27747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:36.724175 27747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:36.725270 27747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:36.725309 27747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:36.725322 27747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:36.725334 27747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:36.725346 27747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:36.726639 27747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:41741
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:41741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:36.726853 27747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:36.727100 27747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:36.732185 27753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:36.732199 27756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:36.732200 27754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:36.732685 27747 server_base.cc:1034] running on GCE node
I20260812 14:26:36.732990 27747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:36.733289 27747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:36.734443 27747 hybrid_clock.cc:648] HybridClock initialized: now 1786544796734420 us; error 42 us; skew 500 ppm
I20260812 14:26:36.735667 27747 webserver.cc:460] Webserver started at http://127.5.148.62:46009/ using document root <none> and password file <none>
I20260812 14:26:36.735927 27747 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:36.736007 27747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:36.736210 27747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:36.737315 27747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "5dc38358d8c642d3bfe391a3f73ba41b"
format_stamp: "Formatted at 2026-08-12 14:26:36 on dist-test-slave-8w9v"
server_key: "ab8b0638ab81526471044aba05457705"
server_key_iv: "6e8a57fb650f5ade184bfa99701daa3d"
server_key_version: "encryptionkey@0"
I20260812 14:26:36.737709 27747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "5dc38358d8c642d3bfe391a3f73ba41b"
format_stamp: "Formatted at 2026-08-12 14:26:36 on dist-test-slave-8w9v"
server_key: "ab8b0638ab81526471044aba05457705"
server_key_iv: "6e8a57fb650f5ade184bfa99701daa3d"
server_key_version: "encryptionkey@0"
I20260812 14:26:36.739059 27747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:26:36.739784 27763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:36.739964 27747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:26:36.740075 27747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "5dc38358d8c642d3bfe391a3f73ba41b"
format_stamp: "Formatted at 2026-08-12 14:26:36 on dist-test-slave-8w9v"
server_key: "ab8b0638ab81526471044aba05457705"
server_key_iv: "6e8a57fb650f5ade184bfa99701daa3d"
server_key_version: "encryptionkey@0"
I20260812 14:26:36.740180 27747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:36.772328 27747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:36.772715 27747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:36.772882 27747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:36.777228 27747 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:41741
I20260812 14:26:36.777275 27821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:41741 every 8 connection(s)
I20260812 14:26:36.777669 27747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:36.778689 27822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:36.781162 27822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b: Bootstrap starting.
I20260812 14:26:36.781788 27822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:36.782105 27822 log.cc:824] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:36.782814 27822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b: No bootstrap required, opened a new log
I20260812 14:26:36.784571 27822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc38358d8c642d3bfe391a3f73ba41b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 41741 } }
I20260812 14:26:36.784718 27822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:36.784740 27822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc38358d8c642d3bfe391a3f73ba41b, State: Initialized, Role: FOLLOWER
I20260812 14:26:36.784921 27822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc38358d8c642d3bfe391a3f73ba41b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 41741 } }
I20260812 14:26:36.784986 27822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:36.785009 27822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:36.785077 27822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:36.785677 27822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc38358d8c642d3bfe391a3f73ba41b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 41741 } }
I20260812 14:26:36.785754 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 27747
I20260812 14:26:36.785801 27822 leader_election.cc:302] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dc38358d8c642d3bfe391a3f73ba41b; no voters:
I20260812 14:26:36.785859 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:36.786011 27822 leader_election.cc:288] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:36.786082 5712 external_mini_cluster.cc:1428] Setting key 81a12c1281ab784e5b2e60902f6f5d2f
I20260812 14:26:36.786140 27825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:36.786339 27822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:36.786360 27825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [term 1 LEADER]: Becoming Leader. State: Replica: 5dc38358d8c642d3bfe391a3f73ba41b, State: Running, Role: LEADER
I20260812 14:26:36.786549 27825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc38358d8c642d3bfe391a3f73ba41b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 41741 } }
I20260812 14:26:36.786998 27826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dc38358d8c642d3bfe391a3f73ba41b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc38358d8c642d3bfe391a3f73ba41b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 41741 } } }
I20260812 14:26:36.787082 27827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dc38358d8c642d3bfe391a3f73ba41b. Latest consensus state: current_term: 1 leader_uuid: "5dc38358d8c642d3bfe391a3f73ba41b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc38358d8c642d3bfe391a3f73ba41b" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 41741 } } }
I20260812 14:26:36.787094 27826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:36.787148 27827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:36.787335 27838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:36.787894 27838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:36.789669 27838 catalog_manager.cc:1347] Generated new cluster ID: d229005aedb9433ab84f2a6d54709453
I20260812 14:26:36.789743 27838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:36.792369 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.786771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.5.148.62:41741)
0812 14:26:36.786990 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:36.787001 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:36.787040 (+ 39us) server_negotiation.cc:407] Connection header received
0812 14:26:36.787176 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:36.787186 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:36.787262 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:36.787424 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:36.787740 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.789009 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.790125 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.790724 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.790845 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.791624 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:36.791628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:36.791628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:36.791831 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:36.791872 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:36.791885 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:36.791956 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:36.792052 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 14:26:36.792151 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260812 14:26:36.797408 27838 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:36.797926 27838 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:36.801663 27838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b: Generated new TSK 0
I20260812 14:26:36.801862 27838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:36.803489 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:41741
--builtin_ntp_servers=127.5.148.20:40369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:26:36.888888 27851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:36.889133 27851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:36.889170 27851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:36.890355 27851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:36.890398 27851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:36.890424 27851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:26:36.891789 27851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:41741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:36.892038 27851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:36.892405 27851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:36.897646 27860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:36.897634 27858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:36.897634 27856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:36.898077 27851 server_base.cc:1034] running on GCE node
I20260812 14:26:36.898309 27851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:36.898569 27851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:36.899732 27851 hybrid_clock.cc:648] HybridClock initialized: now 1786544796899717 us; error 33 us; skew 500 ppm
I20260812 14:26:36.901223 27851 webserver.cc:460] Webserver started at http://127.5.148.1:39251/ using document root <none> and password file <none>
I20260812 14:26:36.901504 27851 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:36.901575 27851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:36.901731 27851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:36.902841 27851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "200e58e902d14601a98ac765d65ece2a"
format_stamp: "Formatted at 2026-08-12 14:26:36 on dist-test-slave-8w9v"
server_key: "89dd823d0e5e7b8e50cdabb9f815f3dc"
server_key_iv: "18967708802a2101aca643ceedb01039"
server_key_version: "encryptionkey@0"
I20260812 14:26:36.903228 27851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "200e58e902d14601a98ac765d65ece2a"
format_stamp: "Formatted at 2026-08-12 14:26:36 on dist-test-slave-8w9v"
server_key: "89dd823d0e5e7b8e50cdabb9f815f3dc"
server_key_iv: "18967708802a2101aca643ceedb01039"
server_key_version: "encryptionkey@0"
I20260812 14:26:36.904644 27851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:26:36.905452 27866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:36.905632 27851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:26:36.905745 27851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "200e58e902d14601a98ac765d65ece2a"
format_stamp: "Formatted at 2026-08-12 14:26:36 on dist-test-slave-8w9v"
server_key: "89dd823d0e5e7b8e50cdabb9f815f3dc"
server_key_iv: "18967708802a2101aca643ceedb01039"
server_key_version: "encryptionkey@0"
I20260812 14:26:36.905825 27851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:36.922734 27851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:36.923081 27851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:36.923249 27851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:36.923516 27851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:36.923883 27851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:36.923919 27851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:36.923943 27851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:36.923985 27851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:36.930538 27851 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:45435
I20260812 14:26:36.930586 28001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:45435 every 8 connection(s)
I20260812 14:26:36.930930 27851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:26:36.936105 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.931659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:33489 (local address 127.5.148.62:41741)
0812 14:26:36.931730 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:36.931732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:36.932023 (+ 291us) server_negotiation.cc:407] Connection header received
0812 14:26:36.932580 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:36.932581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:36.932595 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:36.932651 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:36.933337 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.933832 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.934786 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.934990 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.935043 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.935848 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:36.935850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:36.935850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:36.935857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:36.935870 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:36.935874 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:36.935944 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:36.936005 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 14:26:36.936044 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 14:26:36.936373 28005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:36.931651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:41741 (local address 127.5.148.1:33489)
0812 14:26:36.931837 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:36.931906 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:36.932483 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:36.932719 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:36.932722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:36.932732 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:36.933261 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.933265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.933925 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.933925 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.934707 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:36.934709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:36.935067 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:36.935068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:36.935178 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:36.935698 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:26:36.935698 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:36.935727 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:26:36.935752 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:36.935933 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:36.935938 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:36.935941 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:26:36.936004 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 14:26:36.936085 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260812 14:26:36.936892 28002 heartbeater.cc:347] Connected to a master server at 127.5.148.62:41741
I20260812 14:26:36.936987 28002 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:36.937142 28002 heartbeater.cc:511] Master 127.5.148.62:41741 requested a full tablet report, sending...
I20260812 14:26:36.937674 27781 ts_manager.cc:194] Registered new tserver with Master: 200e58e902d14601a98ac765d65ece2a (127.5.148.1:45435)
I20260812 14:26:36.938310 27781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.148.1:33489
I20260812 14:26:36.939941 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 27851
I20260812 14:26:36.940044 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:26:36.940271 5712 external_mini_cluster.cc:1428] Setting key a3f7a817247451a47ae78193d23fd9f6
I20260812 14:26:36.941087 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:41741
--builtin_ntp_servers=127.5.148.20:40369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:26:37.029256 28006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:37.029487 28006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:37.029511 28006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:37.030660 28006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:37.030697 28006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:37.030723 28006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:26:37.032056 28006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:41741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:37.032426 28006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:37.032804 28006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:37.038405 28014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:37.038444 28015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:37.038437 28017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:37.038687 28006 server_base.cc:1034] running on GCE node
I20260812 14:26:37.039047 28006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:37.039281 28006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:37.040452 28006 hybrid_clock.cc:648] HybridClock initialized: now 1786544797040435 us; error 49 us; skew 500 ppm
I20260812 14:26:37.041823 28006 webserver.cc:460] Webserver started at http://127.5.148.2:42141/ using document root <none> and password file <none>
I20260812 14:26:37.042095 28006 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:37.042171 28006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:37.042330 28006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:37.043524 28006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "3ffdd5202fe44da282aef986eca5089b"
format_stamp: "Formatted at 2026-08-12 14:26:37 on dist-test-slave-8w9v"
server_key: "831b727f2b738f051f6138345586a381"
server_key_iv: "029322ada710d143ee214d41c91f2af8"
server_key_version: "encryptionkey@0"
I20260812 14:26:37.043926 28006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "3ffdd5202fe44da282aef986eca5089b"
format_stamp: "Formatted at 2026-08-12 14:26:37 on dist-test-slave-8w9v"
server_key: "831b727f2b738f051f6138345586a381"
server_key_iv: "029322ada710d143ee214d41c91f2af8"
server_key_version: "encryptionkey@0"
I20260812 14:26:37.045457 28006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:37.046247 28023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:37.046474 28006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:37.046597 28006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "3ffdd5202fe44da282aef986eca5089b"
format_stamp: "Formatted at 2026-08-12 14:26:37 on dist-test-slave-8w9v"
server_key: "831b727f2b738f051f6138345586a381"
server_key_iv: "029322ada710d143ee214d41c91f2af8"
server_key_version: "encryptionkey@0"
I20260812 14:26:37.046686 28006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:37.067175 28006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:37.067596 28006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:37.067783 28006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:37.068090 28006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:37.068543 28006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:37.068606 28006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:37.068655 28006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:37.068703 28006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:37.075577 28006 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:32969
I20260812 14:26:37.075629 28156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:32969 every 8 connection(s)
I20260812 14:26:37.076107 28006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:26:37.077932 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 28006
I20260812 14:26:37.078058 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:26:37.078269 5712 external_mini_cluster.cc:1428] Setting key a93158550159a52f354b121e7fac89ab
I20260812 14:26:37.079253 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:41741
--builtin_ntp_servers=127.5.148.20:40369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 14:26:37.082684 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.076948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:43081 (local address 127.5.148.62:41741)
0812 14:26:37.077052 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:37.077055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:37.077268 (+ 213us) server_negotiation.cc:407] Connection header received
0812 14:26:37.077853 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:37.077854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:37.077865 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:37.077936 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:37.078895 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.079550 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.081095 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.081417 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.081487 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.082410 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:37.082412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:37.082412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:37.082417 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:37.082433 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:37.082437 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:37.082507 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:37.082597 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 14:26:37.082624 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 14:26:37.082986 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.076917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:41741 (local address 127.5.148.2:43081)
0812 14:26:37.077145 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:37.077181 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:37.077734 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:37.078023 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:37.078028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:37.078042 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:37.078802 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:37.078808 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.079647 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.079648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:37.080987 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:37.080990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.081504 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.081505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:37.081627 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.082246 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:26:37.082247 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:37.082276 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:26:37.082300 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:37.082511 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:37.082517 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:37.082521 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:26:37.082592 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 14:26:37.082711 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":97,"threads_started":1}
I20260812 14:26:37.083590 28157 heartbeater.cc:347] Connected to a master server at 127.5.148.62:41741
I20260812 14:26:37.083696 28157 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:37.083938 28157 heartbeater.cc:511] Master 127.5.148.62:41741 requested a full tablet report, sending...
I20260812 14:26:37.084396 27781 ts_manager.cc:194] Registered new tserver with Master: 3ffdd5202fe44da282aef986eca5089b (127.5.148.2:32969)
I20260812 14:26:37.084952 27781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.148.2:43081
W20260812 14:26:37.165400 28161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:37.165643 28161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:37.165668 28161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:37.166800 28161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:37.166836 28161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:37.166860 28161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:26:37.168162 28161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:41741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:37.168378 28161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:37.168627 28161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:37.173738 28168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:37.173808 28173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:37.173728 28171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:37.173934 28161 server_base.cc:1034] running on GCE node
I20260812 14:26:37.174263 28161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:37.174496 28161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:37.175661 28161 hybrid_clock.cc:648] HybridClock initialized: now 1786544797175651 us; error 40 us; skew 500 ppm
I20260812 14:26:37.176916 28161 webserver.cc:460] Webserver started at http://127.5.148.3:45217/ using document root <none> and password file <none>
I20260812 14:26:37.177156 28161 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:37.177224 28161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:37.177377 28161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:37.178462 28161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "f8d95de3863f42b09ac61f2c2ae44fd9"
format_stamp: "Formatted at 2026-08-12 14:26:37 on dist-test-slave-8w9v"
server_key: "ecd72e4749bf9b9ae99061243b57de4b"
server_key_iv: "0d082e63abe261b13533184b08da32bb"
server_key_version: "encryptionkey@0"
I20260812 14:26:37.178833 28161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "f8d95de3863f42b09ac61f2c2ae44fd9"
format_stamp: "Formatted at 2026-08-12 14:26:37 on dist-test-slave-8w9v"
server_key: "ecd72e4749bf9b9ae99061243b57de4b"
server_key_iv: "0d082e63abe261b13533184b08da32bb"
server_key_version: "encryptionkey@0"
I20260812 14:26:37.180281 28161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:26:37.181037 28183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:37.181206 28161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:26:37.181308 28161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "f8d95de3863f42b09ac61f2c2ae44fd9"
format_stamp: "Formatted at 2026-08-12 14:26:37 on dist-test-slave-8w9v"
server_key: "ecd72e4749bf9b9ae99061243b57de4b"
server_key_iv: "0d082e63abe261b13533184b08da32bb"
server_key_version: "encryptionkey@0"
I20260812 14:26:37.181420 28161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:37.206382 28161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:37.206679 28161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:37.206785 28161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:37.207012 28161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:37.207376 28161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:37.207412 28161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:37.207455 28161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:37.207481 28161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:37.213599 28161 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:40993
I20260812 14:26:37.213652 28325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:40993 every 8 connection(s)
I20260812 14:26:37.214001 28161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:26:37.216347 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 28161
I20260812 14:26:37.216447 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:26:37.216651 5712 external_mini_cluster.cc:1428] Setting key c6fd046d6395b1b0c3ba4b0e117df461
I20260812 14:26:37.219834 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.214761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:43275 (local address 127.5.148.62:41741)
0812 14:26:37.214834 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:37.214836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:37.215084 (+ 248us) server_negotiation.cc:407] Connection header received
0812 14:26:37.215663 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:37.215664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:37.215678 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:37.215733 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:37.216588 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.217303 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.218366 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.218583 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.218661 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.219553 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:37.219555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:37.219555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:37.219561 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:37.219574 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:37.219578 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:37.219623 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:37.219726 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 14:26:37.219756 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:26:37.220070 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.214755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:41741 (local address 127.5.148.3:43275)
0812 14:26:37.214957 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:37.214993 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:37.215548 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:37.215805 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:37.215809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:37.215819 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:37.216477 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:37.216486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.217382 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.217384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:37.218259 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:37.218261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.218670 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.218672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:37.218809 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.219402 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:26:37.219403 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:37.219436 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:26:37.219459 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:37.219634 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:37.219639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:37.219642 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:26:37.219727 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 14:26:37.219828 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260812 14:26:37.220559 28327 heartbeater.cc:347] Connected to a master server at 127.5.148.62:41741
I20260812 14:26:37.220643 28327 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:37.220847 28327 heartbeater.cc:511] Master 127.5.148.62:41741 requested a full tablet report, sending...
I20260812 14:26:37.221207 27781 ts_manager.cc:194] Registered new tserver with Master: f8d95de3863f42b09ac61f2c2ae44fd9 (127.5.148.3:40993)
I20260812 14:26:37.221583 27781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.148.3:43275
I20260812 14:26:37.228588 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:26:37.233207 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.230070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.5.148.62:41741)
0812 14:26:37.230131 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:37.230133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:37.230264 (+ 131us) server_negotiation.cc:407] Connection header received
0812 14:26:37.230343 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:37.230344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:37.230358 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:37.230433 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:37.230634 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.231109 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.231874 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.232074 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.232197 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.232896 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:37.232897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:37.232898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:37.232902 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:37.232925 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:37.232929 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:37.233038 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:37.233106 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 14:26:37.233117 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:26:37.234683 27781 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:26:37.236366 27781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:26:37.243124 28337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.238772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:40993 (local address 127.0.0.1:37724)
0812 14:26:37.239249 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:37.239293 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:37.239368 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:37.239682 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:37.239685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:37.239688 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:37.239768 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:37.239770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.240838 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.240839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:37.242115 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:37.242118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.242819 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.242820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:37.242897 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.242910 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 14:26:37.242988 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 14:26:37.243044 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":40,"thread_start_us":67,"threads_started":1}
I20260812 14:26:37.243136 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.238868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.5.148.3:40993)
0812 14:26:37.239089 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:37.239095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:37.239372 (+ 277us) server_negotiation.cc:407] Connection header received
0812 14:26:37.239479 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:37.239482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:37.239500 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:37.239590 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:37.239899 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.240719 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.242223 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.242727 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.242818 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.242920 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:37.243003 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 14:26:37.243044 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":137,"threads_started":1}
I20260812 14:26:37.243371 28336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.238773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:32969 (local address 127.0.0.1:48916)
0812 14:26:37.239457 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:37.239465 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:37.239533 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:37.239805 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:37.239806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:37.239809 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:37.239846 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:37.239847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.240780 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.240781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:37.241927 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:37.241930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.242977 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.242978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:37.243038 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.243042 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:26:37.243087 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:26:37.243131 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":41,"thread_start_us":131,"threads_started":1}
I20260812 14:26:37.243538 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.239066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48916 (local address 127.5.148.2:32969)
0812 14:26:37.239241 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:37.239245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:37.239536 (+ 291us) server_negotiation.cc:407] Connection header received
0812 14:26:37.239605 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:37.239607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:37.239621 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:37.239906 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:37.239990 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.240692 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.242022 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.242793 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.242863 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.242943 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:37.243172 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 14:26:37.243381 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260812 14:26:37.244372 28241 tablet_service.cc:1467] Processing CreateTablet for tablet 7325a98c37f14ad4bb57f358f977e379 (DEFAULT_TABLE table=test-table [id=b6bf568d0c834cb696e6da4c67c4101b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:37.244554 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.238792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:45435 (local address 127.0.0.1:55088)
0812 14:26:37.239618 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:37.239626 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:37.239668 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:37.240845 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:37.240846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:37.240849 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:37.240886 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:37.240887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.242175 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.242176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:37.243246 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:37.243249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.244387 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.244388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:37.244432 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.244435 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:26:37.244479 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 14:26:37.244505 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":117,"threads_started":1}
I20260812 14:26:37.244709 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.239055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.5.148.1:45435)
0812 14:26:37.239229 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:37.239233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:37.239730 (+ 497us) server_negotiation.cc:407] Connection header received
0812 14:26:37.239791 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:37.239794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:37.239810 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:37.239892 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:37.240976 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.242103 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.243345 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.244196 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.244264 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.244348 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:37.244525 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 14:26:37.244589 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260812 14:26:37.244793 28241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7325a98c37f14ad4bb57f358f977e379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:37.245401 27926 tablet_service.cc:1467] Processing CreateTablet for tablet 7325a98c37f14ad4bb57f358f977e379 (DEFAULT_TABLE table=test-table [id=b6bf568d0c834cb696e6da4c67c4101b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:37.245774 27926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7325a98c37f14ad4bb57f358f977e379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:37.248528 28344 tablet_bootstrap.cc:492] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9: Bootstrap starting.
I20260812 14:26:37.248643 28080 tablet_service.cc:1467] Processing CreateTablet for tablet 7325a98c37f14ad4bb57f358f977e379 (DEFAULT_TABLE table=test-table [id=b6bf568d0c834cb696e6da4c67c4101b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:37.249037 28080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7325a98c37f14ad4bb57f358f977e379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:37.249457 28344 tablet_bootstrap.cc:654] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:37.249882 28344 log.cc:824] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:37.250749 28344 tablet_bootstrap.cc:492] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9: No bootstrap required, opened a new log
I20260812 14:26:37.250825 28344 ts_tablet_manager.cc:1397] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:26:37.252403 28345 tablet_bootstrap.cc:492] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a: Bootstrap starting.
I20260812 14:26:37.252792 28347 tablet_bootstrap.cc:492] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b: Bootstrap starting.
I20260812 14:26:37.253163 28345 tablet_bootstrap.cc:654] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:37.253536 28347 tablet_bootstrap.cc:654] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:37.253522 28345 log.cc:824] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:37.253486 28344 raft_consensus.cc:348] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200e58e902d14601a98ac765d65ece2a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45435 } } peers { permanent_uuid: "f8d95de3863f42b09ac61f2c2ae44fd9" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40993 } } peers { permanent_uuid: "3ffdd5202fe44da282aef986eca5089b" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 32969 } }
I20260812 14:26:37.253664 28344 raft_consensus.cc:374] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:37.253706 28344 raft_consensus.cc:729] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d95de3863f42b09ac61f2c2ae44fd9, State: Initialized, Role: FOLLOWER
I20260812 14:26:37.253829 28344 consensus_queue.cc:260] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200e58e902d14601a98ac765d65ece2a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45435 } } peers { permanent_uuid: "f8d95de3863f42b09ac61f2c2ae44fd9" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40993 } } peers { permanent_uuid: "3ffdd5202fe44da282aef986eca5089b" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 32969 } }
I20260812 14:26:37.253866 28347 log.cc:824] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:37.254153 28344 ts_tablet_manager.cc:1428] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:26:37.254180 28327 heartbeater.cc:503] Master 127.5.148.62:41741 was elected leader, sending a full tablet report...
I20260812 14:26:37.254976 28345 tablet_bootstrap.cc:492] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a: No bootstrap required, opened a new log
I20260812 14:26:37.255054 28345 ts_tablet_manager.cc:1397] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:26:37.255069 28347 tablet_bootstrap.cc:492] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b: No bootstrap required, opened a new log
I20260812 14:26:37.255133 28347 ts_tablet_manager.cc:1397] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:26:37.257053 28345 raft_consensus.cc:348] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200e58e902d14601a98ac765d65ece2a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45435 } } peers { permanent_uuid: "f8d95de3863f42b09ac61f2c2ae44fd9" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40993 } } peers { permanent_uuid: "3ffdd5202fe44da282aef986eca5089b" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 32969 } }
I20260812 14:26:37.257177 28345 raft_consensus.cc:374] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:37.257200 28345 raft_consensus.cc:729] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200e58e902d14601a98ac765d65ece2a, State: Initialized, Role: FOLLOWER
I20260812 14:26:37.257138 28347 raft_consensus.cc:348] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200e58e902d14601a98ac765d65ece2a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45435 } } peers { permanent_uuid: "f8d95de3863f42b09ac61f2c2ae44fd9" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40993 } } peers { permanent_uuid: "3ffdd5202fe44da282aef986eca5089b" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 32969 } }
I20260812 14:26:37.257334 28347 raft_consensus.cc:374] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:37.257385 28347 raft_consensus.cc:729] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffdd5202fe44da282aef986eca5089b, State: Initialized, Role: FOLLOWER
I20260812 14:26:37.257342 28345 consensus_queue.cc:260] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200e58e902d14601a98ac765d65ece2a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45435 } } peers { permanent_uuid: "f8d95de3863f42b09ac61f2c2ae44fd9" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40993 } } peers { permanent_uuid: "3ffdd5202fe44da282aef986eca5089b" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 32969 } }
I20260812 14:26:37.257505 28347 consensus_queue.cc:260] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200e58e902d14601a98ac765d65ece2a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45435 } } peers { permanent_uuid: "f8d95de3863f42b09ac61f2c2ae44fd9" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40993 } } peers { permanent_uuid: "3ffdd5202fe44da282aef986eca5089b" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 32969 } }
I20260812 14:26:37.257627 28345 ts_tablet_manager.cc:1428] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:26:37.257671 28002 heartbeater.cc:503] Master 127.5.148.62:41741 was elected leader, sending a full tablet report...
I20260812 14:26:37.257797 28347 ts_tablet_manager.cc:1428] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 14:26:37.257856 28157 heartbeater.cc:503] Master 127.5.148.62:41741 was elected leader, sending a full tablet report...
I20260812 14:26:37.290041 28349 raft_consensus.cc:482] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:26:37.290171 28349 raft_consensus.cc:504] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200e58e902d14601a98ac765d65ece2a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45435 } } peers { permanent_uuid: "f8d95de3863f42b09ac61f2c2ae44fd9" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40993 } } peers { permanent_uuid: "3ffdd5202fe44da282aef986eca5089b" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 32969 } }
I20260812 14:26:37.290493 28349 leader_election.cc:288] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 200e58e902d14601a98ac765d65ece2a (127.5.148.1:45435), 3ffdd5202fe44da282aef986eca5089b (127.5.148.2:32969)
I20260812 14:26:37.293490 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.290689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:32969 (local address 127.5.148.3:45529)
0812 14:26:37.290753 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:37.290767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:37.290825 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:37.291029 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:37.291030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:37.291032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:37.291112 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:37.291114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.291801 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.291802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:37.292674 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:37.292677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.293238 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.293240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:37.293325 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.293331 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 14:26:37.293385 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 14:26:37.293432 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 14:26:37.293530 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.290809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:45529 (local address 127.5.148.2:32969)
0812 14:26:37.290858 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:37.290860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:37.290869 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:26:37.290900 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:37.290901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:37.290932 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:37.291004 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:37.291191 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.291724 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.292763 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.293161 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.293217 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.293225 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:37.293400 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 14:26:37.293444 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:26:37.293672 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.290702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:45435 (local address 127.5.148.3:47875)
0812 14:26:37.290941 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:37.290951 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:37.291005 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:37.291138 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:37.291139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:37.291141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:37.291239 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:37.291241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.291870 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.291871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:37.292791 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:37.292793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.293380 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.293381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:37.293476 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.293480 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:26:37.293538 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 14:26:37.293578 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":52,"thread_start_us":126,"threads_started":1}
I20260812 14:26:37.293696 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.290805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:47875 (local address 127.5.148.1:45435)
0812 14:26:37.290852 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:37.290854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:37.291012 (+ 158us) server_negotiation.cc:407] Connection header received
0812 14:26:37.291051 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:37.291052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:37.291063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:37.291128 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:37.291311 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.291791 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.292884 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.293280 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.293351 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.293361 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:37.293570 (+ 209us) server_negotiation.cc:299] Negotiation successful
0812 14:26:37.293620 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:26:37.293910 28105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7325a98c37f14ad4bb57f358f977e379" candidate_uuid: "f8d95de3863f42b09ac61f2c2ae44fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ffdd5202fe44da282aef986eca5089b" is_pre_election: true
I20260812 14:26:37.294111 28105 raft_consensus.cc:2454] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8d95de3863f42b09ac61f2c2ae44fd9 in term 0.
I20260812 14:26:37.294086 27952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7325a98c37f14ad4bb57f358f977e379" candidate_uuid: "f8d95de3863f42b09ac61f2c2ae44fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "200e58e902d14601a98ac765d65ece2a" is_pre_election: true
I20260812 14:26:37.294248 27952 raft_consensus.cc:2454] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8d95de3863f42b09ac61f2c2ae44fd9 in term 0.
I20260812 14:26:37.294360 28189 leader_election.cc:302] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ffdd5202fe44da282aef986eca5089b, f8d95de3863f42b09ac61f2c2ae44fd9; no voters:
I20260812 14:26:37.294518 28349 raft_consensus.cc:2781] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:26:37.294579 28349 raft_consensus.cc:482] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:26:37.294601 28349 raft_consensus.cc:3037] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:37.295323 28349 raft_consensus.cc:504] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200e58e902d14601a98ac765d65ece2a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45435 } } peers { permanent_uuid: "f8d95de3863f42b09ac61f2c2ae44fd9" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40993 } } peers { permanent_uuid: "3ffdd5202fe44da282aef986eca5089b" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 32969 } }
I20260812 14:26:37.295480 28349 leader_election.cc:288] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [CANDIDATE]: Term 1 election: Requested vote from peers 200e58e902d14601a98ac765d65ece2a (127.5.148.1:45435), 3ffdd5202fe44da282aef986eca5089b (127.5.148.2:32969)
I20260812 14:26:37.295653 27952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7325a98c37f14ad4bb57f358f977e379" candidate_uuid: "f8d95de3863f42b09ac61f2c2ae44fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "200e58e902d14601a98ac765d65ece2a"
I20260812 14:26:37.295680 28105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7325a98c37f14ad4bb57f358f977e379" candidate_uuid: "f8d95de3863f42b09ac61f2c2ae44fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ffdd5202fe44da282aef986eca5089b"
I20260812 14:26:37.295748 28105 raft_consensus.cc:3037] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:37.295742 27952 raft_consensus.cc:3037] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:37.296725 27952 raft_consensus.cc:2454] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8d95de3863f42b09ac61f2c2ae44fd9 in term 1.
I20260812 14:26:37.296747 28105 raft_consensus.cc:2454] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8d95de3863f42b09ac61f2c2ae44fd9 in term 1.
I20260812 14:26:37.296900 28189 leader_election.cc:302] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ffdd5202fe44da282aef986eca5089b, f8d95de3863f42b09ac61f2c2ae44fd9; no voters:
I20260812 14:26:37.297008 28349 raft_consensus.cc:2781] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:37.297178 28349 raft_consensus.cc:686] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [term 1 LEADER]: Becoming Leader. State: Replica: f8d95de3863f42b09ac61f2c2ae44fd9, State: Running, Role: LEADER
I20260812 14:26:37.297333 28349 consensus_queue.cc:237] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200e58e902d14601a98ac765d65ece2a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45435 } } peers { permanent_uuid: "f8d95de3863f42b09ac61f2c2ae44fd9" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40993 } } peers { permanent_uuid: "3ffdd5202fe44da282aef986eca5089b" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 32969 } }
I20260812 14:26:37.298316 27780 catalog_manager.cc:5495] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8d95de3863f42b09ac61f2c2ae44fd9 (127.5.148.3). New cstate: current_term: 1 leader_uuid: "f8d95de3863f42b09ac61f2c2ae44fd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200e58e902d14601a98ac765d65ece2a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8d95de3863f42b09ac61f2c2ae44fd9" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 40993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ffdd5202fe44da282aef986eca5089b" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 32969 } health_report { overall_health: UNKNOWN } } }
I20260812 14:26:37.316906 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.314660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.5.148.3:40993)
0812 14:26:37.314710 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:37.314712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:37.314719 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:26:37.314756 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:37.314756 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:37.314780 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:37.314846 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:26:37.314975 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.315468 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.316230 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:37.316434 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:37.316491 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:37.316622 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:26:37.316711 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:26:37.316793 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:37.316815 (+ 22us) server_negotiation.cc:299] Negotiation successful
0812 14:26:37.316823 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:26:37.318954 27952 raft_consensus.cc:1261] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a [term 1 FOLLOWER]: Refusing update from remote peer f8d95de3863f42b09ac61f2c2ae44fd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:26:37.318961 28105 raft_consensus.cc:1261] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b [term 1 FOLLOWER]: Refusing update from remote peer f8d95de3863f42b09ac61f2c2ae44fd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:26:37.319380 28354 consensus_queue.cc:1035] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ffdd5202fe44da282aef986eca5089b" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:26:37.319537 28349 consensus_queue.cc:1035] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e58e902d14601a98ac765d65ece2a" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:26:37.323364 28361 mvcc.cc:205] Tried to move back new op lower bound from 7317687489815654400 to 7317687489730846720. Current Snapshot: MvccSnapshot[applied={T|T < 7317687489815654400 or (T in {7317687489815654400})}]
I20260812 14:26:37.324575 27781 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:26:37.324687 27781 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:26:37.325459 27781 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5dc38358d8c642d3bfe391a3f73ba41b: Sending DeleteTablet for 3 replicas of tablet 7325a98c37f14ad4bb57f358f977e379
I20260812 14:26:37.325789 28241 tablet_service.cc:1514] Processing DeleteTablet for tablet 7325a98c37f14ad4bb57f358f977e379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:37 UTC) from {username='slave'} at 127.0.0.1:37724
I20260812 14:26:37.325845 27926 tablet_service.cc:1514] Processing DeleteTablet for tablet 7325a98c37f14ad4bb57f358f977e379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:37 UTC) from {username='slave'} at 127.0.0.1:55088
I20260812 14:26:37.326047 28373 tablet_replica.cc:330] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9: stopping tablet replica
I20260812 14:26:37.326028 28080 tablet_service.cc:1514] Processing DeleteTablet for tablet 7325a98c37f14ad4bb57f358f977e379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:37 UTC) from {username='slave'} at 127.0.0.1:48916
I20260812 14:26:37.326145 28373 raft_consensus.cc:2229] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:26:37.326243 28373 raft_consensus.cc:2258] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:26:37.326252 28374 tablet_replica.cc:330] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a: stopping tablet replica
I20260812 14:26:37.326354 28374 raft_consensus.cc:2229] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:26:37.326483 28374 raft_consensus.cc:2258] T 7325a98c37f14ad4bb57f358f977e379 P 200e58e902d14601a98ac765d65ece2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:26:37.326639 28375 tablet_replica.cc:330] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b: stopping tablet replica
I20260812 14:26:37.326779 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 27851
I20260812 14:26:37.326846 28375 raft_consensus.cc:2229] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:26:37.334444 28375 raft_consensus.cc:2258] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:26:37.335209 28375 ts_tablet_manager.cc:1905] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:26:37.335937 28373 ts_tablet_manager.cc:1905] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:26:37.337047 28375 ts_tablet_manager.cc:1918] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:26:37.337131 28375 log.cc:1196] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/7325a98c37f14ad4bb57f358f977e379
I20260812 14:26:37.337400 28375 ts_tablet_manager.cc:1939] T 7325a98c37f14ad4bb57f358f977e379 P 3ffdd5202fe44da282aef986eca5089b: Deleting consensus metadata
I20260812 14:26:37.338011 28373 ts_tablet_manager.cc:1918] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:26:37.338236 27767 catalog_manager.cc:4841] TS 3ffdd5202fe44da282aef986eca5089b (127.5.148.2:32969): tablet 7325a98c37f14ad4bb57f358f977e379 (table test-table [id=b6bf568d0c834cb696e6da4c67c4101b]) successfully deleted
I20260812 14:26:37.338342 28373 log.cc:1196] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/7325a98c37f14ad4bb57f358f977e379
I20260812 14:26:37.338634 28373 ts_tablet_manager.cc:1939] T 7325a98c37f14ad4bb57f358f977e379 P f8d95de3863f42b09ac61f2c2ae44fd9: Deleting consensus metadata
I20260812 14:26:37.339116 27765 catalog_manager.cc:4841] TS f8d95de3863f42b09ac61f2c2ae44fd9 (127.5.148.3:40993): tablet 7325a98c37f14ad4bb57f358f977e379 (table test-table [id=b6bf568d0c834cb696e6da4c67c4101b]) successfully deleted
W20260812 14:26:37.341394 27764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:45435 (error 108)
I20260812 14:26:37.341595 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 28006
I20260812 14:26:37.345659 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:37.341760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:45435 (local address 127.0.0.1:55104)
0812 14:26:37.345556 (+ 3796us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:37.345619 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3759}
W20260812 14:26:37.345834 27764 catalog_manager.cc:4584] TS 200e58e902d14601a98ac765d65ece2a (127.5.148.1:45435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7325a98c37f14ad4bb57f358f977e379: Network error: Client connection negotiation failed: client connection to 127.5.148.1:45435: connect: Connection refused (error 111)
I20260812 14:26:37.358950 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 28161
I20260812 14:26:37.366899 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 27747
2026-08-12T14:26:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (758 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:37 dist-test-slave-8w9v krb5kdc[28379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:37 dist-test-slave-8w9v krb5kdc[28379](info): set up 2 sockets
Aug 12 14:26:37 dist-test-slave-8w9v krb5kdc[28379](info): commencing operation
krb5kdc: starting...
W20260812 14:26:38.500658 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:38 dist-test-slave-8w9v krb5kdc[28379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:26:38.627357 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:39069
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:39069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:38.712371 28394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:38.712599 28394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:38.712623 28394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:38.713795 28394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:38.713840 28394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:38.713853 28394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:38.713891 28394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:38.713922 28394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:38.715394 28394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:39069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:39069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:38.715667 28394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:38.715909 28394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:26:38.720953 28394 server_base.cc:1034] running on GCE node
W20260812 14:26:38.721074 28403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:38.721055 28400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:38.721119 28401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:38.721535 28394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:38.721845 28394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:38.723019 28394 hybrid_clock.cc:648] HybridClock initialized: now 1786544798723007 us; error 45 us; skew 500 ppm
Aug 12 14:26:38 dist-test-slave-8w9v krb5kdc[28379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544798, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:38.729503 28394 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:26:38.730023 28394 webserver.cc:460] Webserver started at http://127.5.148.62:33751/ using document root <none> and password file <none>
I20260812 14:26:38.730259 28394 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:38.730329 28394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:38.730491 28394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:38.731602 28394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "c6a71620e0a44735ae593d64ffcc3788"
format_stamp: "Formatted at 2026-08-12 14:26:38 on dist-test-slave-8w9v"
server_key: "f40180859d5229ffebbd31ee0c6ca3f3"
server_key_iv: "e8e08b30cabfa09ebb30e08470be55f1"
server_key_version: "encryptionkey@0"
I20260812 14:26:38.732062 28394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "c6a71620e0a44735ae593d64ffcc3788"
format_stamp: "Formatted at 2026-08-12 14:26:38 on dist-test-slave-8w9v"
server_key: "f40180859d5229ffebbd31ee0c6ca3f3"
server_key_iv: "e8e08b30cabfa09ebb30e08470be55f1"
server_key_version: "encryptionkey@0"
I20260812 14:26:38.733596 28394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:38.734321 28411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:38.734521 28394 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:26:38.734628 28394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "c6a71620e0a44735ae593d64ffcc3788"
format_stamp: "Formatted at 2026-08-12 14:26:38 on dist-test-slave-8w9v"
server_key: "f40180859d5229ffebbd31ee0c6ca3f3"
server_key_iv: "e8e08b30cabfa09ebb30e08470be55f1"
server_key_version: "encryptionkey@0"
I20260812 14:26:38.734719 28394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:38.753262 28394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:38.755825 28394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:38.756014 28394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:38.760447 28394 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:39069
I20260812 14:26:38.760504 28469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:39069 every 8 connection(s)
I20260812 14:26:38.760913 28394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:38.761795 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:38.763077 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 28394
I20260812 14:26:38.763170 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:38.763403 5712 external_mini_cluster.cc:1428] Setting key de2baaafb77803d5c1971bc4264689d9
I20260812 14:26:38.764875 28470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788: Bootstrap starting.
I20260812 14:26:38.765635 28470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:38.766019 28470 log.cc:824] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:38.766772 28470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788: No bootstrap required, opened a new log
I20260812 14:26:38.769205 28470 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a71620e0a44735ae593d64ffcc3788" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39069 } }
I20260812 14:26:38.769397 28470 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:38.769438 28470 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a71620e0a44735ae593d64ffcc3788, State: Initialized, Role: FOLLOWER
I20260812 14:26:38.769579 28470 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a71620e0a44735ae593d64ffcc3788" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39069 } }
I20260812 14:26:38.769668 28470 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:38.769706 28470 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:38.769763 28470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:38.770553 28470 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a71620e0a44735ae593d64ffcc3788" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39069 } }
I20260812 14:26:38.770649 28470 leader_election.cc:302] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6a71620e0a44735ae593d64ffcc3788; no voters:
I20260812 14:26:38.770843 28470 leader_election.cc:288] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:38.770910 28476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [term 1 FOLLOWER]: Leader election won for term 1
Aug 12 14:26:38 dist-test-slave-8w9v krb5kdc[28379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:26:38.771078 28476 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [term 1 LEADER]: Becoming Leader. State: Replica: c6a71620e0a44735ae593d64ffcc3788, State: Running, Role: LEADER
I20260812 14:26:38.771157 28470 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:38.771190 28476 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a71620e0a44735ae593d64ffcc3788" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39069 } }
I20260812 14:26:38.771512 28476 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6a71620e0a44735ae593d64ffcc3788. Latest consensus state: current_term: 1 leader_uuid: "c6a71620e0a44735ae593d64ffcc3788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a71620e0a44735ae593d64ffcc3788" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39069 } } }
I20260812 14:26:38.771524 28479 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6a71620e0a44735ae593d64ffcc3788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a71620e0a44735ae593d64ffcc3788" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 39069 } } }
I20260812 14:26:38.771598 28476 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:38.771634 28479 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:38.772336 28483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:38.773485 28483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:38.775792 28473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:38.763988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.5.148.62:39069)
0812 14:26:38.764219 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:38.764229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:38.764266 (+ 37us) server_negotiation.cc:407] Connection header received
0812 14:26:38.764836 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:38.764845 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:38.764918 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:38.765070 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:38.766021 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:38.767082 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:38.768183 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:38.768594 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:38.768679 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:38.771940 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:38.771943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:38.771943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:38.771954 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:38.774151 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:38.774507 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:38.774510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:38.774511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:38.774657 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:38.774905 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:38.774906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:38.774906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:38.775133 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:38.775247 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:38.775408 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 14:26:38.775516 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":93,"threads_started":1}
I20260812 14:26:38.780104 28483 catalog_manager.cc:1347] Generated new cluster ID: a18f0069d9244a69a0b90abbdbc0ed7f
I20260812 14:26:38.780179 28483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:38.786204 28483 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:38.786721 28483 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:38.795600 28483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788: Generated new TSK 0
I20260812 14:26:38.795783 28483 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
Entry for principal kudu/127.5.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab.
I20260812 14:26:38.831401 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:39069
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:38.914541 28498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:38.914740 28498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:38.914764 28498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:38.915933 28498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:38.915977 28498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:38.916016 28498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:26:38.917534 28498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.1.keytab
--principal=kudu/127.5.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:39069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:38.917783 28498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:38.918015 28498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:38.923286 28506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:38.923441 28498 server_base.cc:1034] running on GCE node
W20260812 14:26:38.923295 28508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:38.923295 28505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:38.923738 28498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:38.923995 28498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:38.925148 28498 hybrid_clock.cc:648] HybridClock initialized: now 1786544798925123 us; error 40 us; skew 500 ppm
Aug 12 14:26:38 dist-test-slave-8w9v krb5kdc[28379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544798, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:38.927794 28498 init.cc:377] Logged in from keytab as kudu/127.5.148.1@KRBTEST.COM (short username kudu)
I20260812 14:26:38.928288 28498 webserver.cc:460] Webserver started at http://127.5.148.1:46121/ using document root <none> and password file <none>
I20260812 14:26:38.928547 28498 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:38.928617 28498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:38.928761 28498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:38.929834 28498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "dbfb6cef20d54318b5b8a4e388996d1e"
format_stamp: "Formatted at 2026-08-12 14:26:38 on dist-test-slave-8w9v"
server_key: "89d225b7d218f598b9f1fd2a10c9fbaa"
server_key_iv: "0e885029d4e34165d6caed55080c9389"
server_key_version: "encryptionkey@0"
I20260812 14:26:38.930198 28498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "dbfb6cef20d54318b5b8a4e388996d1e"
format_stamp: "Formatted at 2026-08-12 14:26:38 on dist-test-slave-8w9v"
server_key: "89d225b7d218f598b9f1fd2a10c9fbaa"
server_key_iv: "0e885029d4e34165d6caed55080c9389"
server_key_version: "encryptionkey@0"
I20260812 14:26:38.931535 28498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:26:38.932300 28517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:38.932600 28498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:38.932703 28498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "dbfb6cef20d54318b5b8a4e388996d1e"
format_stamp: "Formatted at 2026-08-12 14:26:38 on dist-test-slave-8w9v"
server_key: "89d225b7d218f598b9f1fd2a10c9fbaa"
server_key_iv: "0e885029d4e34165d6caed55080c9389"
server_key_version: "encryptionkey@0"
I20260812 14:26:38.932777 28498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:38.947891 28498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:38.950603 28498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:38.950798 28498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:38.951076 28498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:38.951455 28498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:38.951519 28498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:38.951567 28498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:38.951602 28498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:38.958339 28498 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:37787
I20260812 14:26:38.958379 28652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:37787 every 8 connection(s)
I20260812 14:26:38.958787 28498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:26:38 dist-test-slave-8w9v krb5kdc[28379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544798, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.1@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:26:38.967411 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 28498
I20260812 14:26:38.967554 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:26:38.967820 5712 external_mini_cluster.cc:1428] Setting key a3f80f9df832dfb293dbd7003ae3d180
I20260812 14:26:38.971103 28473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:38.959544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:45013 (local address 127.5.148.62:39069)
0812 14:26:38.959634 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:38.959637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:38.959793 (+ 156us) server_negotiation.cc:407] Connection header received
0812 14:26:38.960392 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:38.960393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:38.960407 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:38.960457 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:38.961520 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:38.962342 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:38.964034 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:38.964448 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:38.964547 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:38.967451 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:38.967452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:38.967453 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:38.967462 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:38.969280 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:38.969834 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:38.969835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:38.969836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:38.969898 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:38.970185 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:38.970186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:38.970187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:38.970338 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:38.970384 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:38.970836 (+ 452us) server_negotiation.cc:299] Negotiation successful
0812 14:26:38.970950 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 14:26:38.971803 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:38.959491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:39069 (local address 127.5.148.1:45013)
0812 14:26:38.959658 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:38.959698 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:38.960295 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:38.960523 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:38.960526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:38.961123 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:38.961398 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:38.961405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:38.962482 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:38.962483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:38.963897 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:38.963902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:38.964564 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:38.964566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:38.964746 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:38.965371 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:26:38.965380 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:38.967312 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:38.969378 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:38.969397 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:38.969399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:38.969741 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:38.969971 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:38.969972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:38.969972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:38.970113 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:38.970448 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:38.970456 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:38.970637 (+ 181us) client_negotiation.cc:769] Sending connection context
0812 14:26:38.971232 (+ 595us) client_negotiation.cc:240] Negotiation successful
0812 14:26:38.971449 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260812 14:26:38.972677 28653 heartbeater.cc:347] Connected to a master server at 127.5.148.62:39069
I20260812 14:26:38.972826 28653 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:38.973040 28653 heartbeater.cc:511] Master 127.5.148.62:39069 requested a full tablet report, sending...
I20260812 14:26:38.973670 28430 ts_manager.cc:194] Registered new tserver with Master: dbfb6cef20d54318b5b8a4e388996d1e (127.5.148.1:37787)
I20260812 14:26:38.974365 28430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.1@KRBTEST.COM'} at 127.5.148.1:45013
WARNING: no policy specified for kudu/127.5.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
Entry for principal kudu/127.5.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab.
I20260812 14:26:38.995285 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:39069
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:39.079756 28658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:39.080024 28658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:39.080067 28658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:39.081311 28658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:39.081410 28658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:39.081461 28658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:26:39.083026 28658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.2.keytab
--principal=kudu/127.5.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:39069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:39.083384 28658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:39.083737 28658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:39.088991 28664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:39.088987 28667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:39.089002 28663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:39.089334 28658 server_base.cc:1034] running on GCE node
I20260812 14:26:39.089550 28658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:39.089761 28658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:39.090938 28658 hybrid_clock.cc:648] HybridClock initialized: now 1786544799090929 us; error 41 us; skew 500 ppm
Aug 12 14:26:39 dist-test-slave-8w9v krb5kdc[28379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544799, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:39.093510 28658 init.cc:377] Logged in from keytab as kudu/127.5.148.2@KRBTEST.COM (short username kudu)
I20260812 14:26:39.093968 28658 webserver.cc:460] Webserver started at http://127.5.148.2:35979/ using document root <none> and password file <none>
I20260812 14:26:39.094214 28658 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:39.094286 28658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:39.094445 28658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:39.095510 28658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "f92100d9818c44609045097967dfb3b3"
format_stamp: "Formatted at 2026-08-12 14:26:39 on dist-test-slave-8w9v"
server_key: "3d06fed527c3685cac5d9f3a566cf4aa"
server_key_iv: "85da624ad0b14a42ccfc2b18becd22cf"
server_key_version: "encryptionkey@0"
I20260812 14:26:39.095888 28658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "f92100d9818c44609045097967dfb3b3"
format_stamp: "Formatted at 2026-08-12 14:26:39 on dist-test-slave-8w9v"
server_key: "3d06fed527c3685cac5d9f3a566cf4aa"
server_key_iv: "85da624ad0b14a42ccfc2b18becd22cf"
server_key_version: "encryptionkey@0"
I20260812 14:26:39.097321 28658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:26:39.098074 28674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:39.098254 28658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:26:39.098368 28658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "f92100d9818c44609045097967dfb3b3"
format_stamp: "Formatted at 2026-08-12 14:26:39 on dist-test-slave-8w9v"
server_key: "3d06fed527c3685cac5d9f3a566cf4aa"
server_key_iv: "85da624ad0b14a42ccfc2b18becd22cf"
server_key_version: "encryptionkey@0"
I20260812 14:26:39.098448 28658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:39.117009 28658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:39.119086 28658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:39.119264 28658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:39.119560 28658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:39.119935 28658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:39.119993 28658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:39.120039 28658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:39.120067 28658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:39.126358 28658 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:35147
I20260812 14:26:39.126405 28807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:35147 every 8 connection(s)
I20260812 14:26:39.126740 28658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:26:39.131062 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 28658
I20260812 14:26:39.131183 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:26:39.131393 5712 external_mini_cluster.cc:1428] Setting key 172cd4ff0de942768677b5107c46de80
Aug 12 14:26:39 dist-test-slave-8w9v krb5kdc[28379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544799, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.2@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:26:39.136754 28473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.127497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:55051 (local address 127.5.148.62:39069)
0812 14:26:39.127564 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:39.127566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:39.127782 (+ 216us) server_negotiation.cc:407] Connection header received
0812 14:26:39.128377 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:39.128378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:39.128402 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:39.128460 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:39.129422 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.129916 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.130959 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.131181 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.131270 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.133698 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:39.133699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:39.133700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:39.133707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:39.135249 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:39.135653 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:39.135655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:39.135655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:39.135717 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:39.135981 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:39.135982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:39.135983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:39.136109 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:39.136211 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:39.136623 (+ 412us) server_negotiation.cc:299] Negotiation successful
0812 14:26:39.136691 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:26:39.137147 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.127467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:39069 (local address 127.5.148.2:55051)
0812 14:26:39.127646 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:39.127689 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:39.128284 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:39.128527 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:39.128530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:39.129120 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:39.129321 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:39.129325 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.130003 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.130004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:39.130870 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:39.130873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.131310 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.131311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:39.131477 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.132061 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:26:39.132071 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:39.133600 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:39.135338 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:39.135346 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:39.135348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:39.135575 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:39.135795 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:39.135796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:39.135796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:39.135915 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:39.136207 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:39.136215 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:39.136473 (+ 258us) client_negotiation.cc:769] Sending connection context
0812 14:26:39.136609 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 14:26:39.136857 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260812 14:26:39.137663 28808 heartbeater.cc:347] Connected to a master server at 127.5.148.62:39069
I20260812 14:26:39.137784 28808 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:39.138033 28808 heartbeater.cc:511] Master 127.5.148.62:39069 requested a full tablet report, sending...
I20260812 14:26:39.138500 28430 ts_manager.cc:194] Registered new tserver with Master: f92100d9818c44609045097967dfb3b3 (127.5.148.2:35147)
I20260812 14:26:39.138883 28430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.2@KRBTEST.COM'} at 127.5.148.2:55051
WARNING: no policy specified for kudu/127.5.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
Entry for principal kudu/127.5.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab.
I20260812 14:26:39.159636 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:39069
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:39.245034 28814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:39.245251 28814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:39.245277 28814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:39.246431 28814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:39.246477 28814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:39.246505 28814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:26:39.247972 28814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.3.keytab
--principal=kudu/127.5.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:39069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:39.248242 28814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:39.248462 28814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:39.253729 28819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:39.253748 28820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:39.253775 28823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:39.253981 28814 server_base.cc:1034] running on GCE node
I20260812 14:26:39.254225 28814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:39.254521 28814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:39.255677 28814 hybrid_clock.cc:648] HybridClock initialized: now 1786544799255655 us; error 34 us; skew 500 ppm
Aug 12 14:26:39 dist-test-slave-8w9v krb5kdc[28379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544799, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:39.258399 28814 init.cc:377] Logged in from keytab as kudu/127.5.148.3@KRBTEST.COM (short username kudu)
I20260812 14:26:39.258895 28814 webserver.cc:460] Webserver started at http://127.5.148.3:39985/ using document root <none> and password file <none>
I20260812 14:26:39.259152 28814 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:39.259225 28814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:39.259375 28814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:39.260547 28814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "073f18bcf5e44c4bae6716000ed3932b"
format_stamp: "Formatted at 2026-08-12 14:26:39 on dist-test-slave-8w9v"
server_key: "c67b713740c779fdd603c88afa8867da"
server_key_iv: "27bc453e4e62afecfade4d5f2277d516"
server_key_version: "encryptionkey@0"
I20260812 14:26:39.260968 28814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "073f18bcf5e44c4bae6716000ed3932b"
format_stamp: "Formatted at 2026-08-12 14:26:39 on dist-test-slave-8w9v"
server_key: "c67b713740c779fdd603c88afa8867da"
server_key_iv: "27bc453e4e62afecfade4d5f2277d516"
server_key_version: "encryptionkey@0"
I20260812 14:26:39.262284 28814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:26:39.263017 28834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:39.263188 28814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:26:39.263280 28814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "073f18bcf5e44c4bae6716000ed3932b"
format_stamp: "Formatted at 2026-08-12 14:26:39 on dist-test-slave-8w9v"
server_key: "c67b713740c779fdd603c88afa8867da"
server_key_iv: "27bc453e4e62afecfade4d5f2277d516"
server_key_version: "encryptionkey@0"
I20260812 14:26:39.263367 28814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:39.314152 28814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:39.316288 28814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:39.316473 28814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:39.316737 28814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:39.317072 28814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:39.317129 28814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:39.317173 28814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:39.317217 28814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:39.323387 28814 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:34991
I20260812 14:26:39.323460 28976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:34991 every 8 connection(s)
I20260812 14:26:39.323791 28814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:26:39.326175 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 28814
I20260812 14:26:39.326287 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:26:39.326481 5712 external_mini_cluster.cc:1428] Setting key ec515b1d6aed53d7fc29e2a0d0a24df0
Aug 12 14:26:39 dist-test-slave-8w9v krb5kdc[28379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544799, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.3@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:26:39.333401 28473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.324522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:51619 (local address 127.5.148.62:39069)
0812 14:26:39.324567 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:39.324570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:39.324796 (+ 226us) server_negotiation.cc:407] Connection header received
0812 14:26:39.325280 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:39.325281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:39.325290 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:39.325352 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:39.326638 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.327222 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.328302 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.328508 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.328568 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.330743 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:39.330744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:39.330745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:39.330751 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:39.332051 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:39.332462 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:39.332463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:39.332463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:39.332517 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:39.332730 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:39.332731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:39.332731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:39.332867 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:39.332923 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:39.333244 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 14:26:39.333316 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:26:39.333841 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.324484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:39069 (local address 127.5.148.3:51619)
0812 14:26:39.324661 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:39.324705 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:39.325198 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:39.325483 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:39.325487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:39.326182 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:39.326500 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:39.326507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.327319 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.327320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:39.328231 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:39.328234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.328586 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.328587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:39.328688 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.329247 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:26:39.329256 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:39.330664 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:39.332163 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:39.332173 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:39.332175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:39.332396 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:39.332578 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:26:39.332578 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:26:39.332579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:26:39.332676 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:26:39.332924 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:39.332929 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:39.333040 (+ 111us) client_negotiation.cc:769] Sending connection context
0812 14:26:39.333484 (+ 444us) client_negotiation.cc:240] Negotiation successful
0812 14:26:39.333680 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260812 14:26:39.334309 28977 heartbeater.cc:347] Connected to a master server at 127.5.148.62:39069
I20260812 14:26:39.334404 28977 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:39.334549 28977 heartbeater.cc:511] Master 127.5.148.62:39069 requested a full tablet report, sending...
I20260812 14:26:39.334937 28430 ts_manager.cc:194] Registered new tserver with Master: 073f18bcf5e44c4bae6716000ed3932b (127.5.148.3:34991)
I20260812 14:26:39.335268 28430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.148.3@KRBTEST.COM'} at 127.5.148.3:51619
I20260812 14:26:39.338111 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:26:39.345331 28473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.339581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.5.148.62:39069)
0812 14:26:39.339618 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:39.339620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:39.339774 (+ 154us) server_negotiation.cc:407] Connection header received
0812 14:26:39.339813 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:39.339814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:39.339823 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:39.339873 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:39.340504 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.340959 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.341624 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.341801 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.341853 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.342948 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:39.342949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:39.342949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:39.342956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:39.344407 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:39.344697 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:39.344698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:39.344698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:39.344746 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:39.344909 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:39.344909 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:39.344910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:39.345044 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:39.345121 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:39.345217 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 14:26:39.345272 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:26:39.347054 28430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:26:39.348837 28430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:26:39.355448 28996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.351481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46534 (local address 127.5.148.3:34991)
0812 14:26:39.351687 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:39.351692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:39.351706 (+ 14us) server_negotiation.cc:407] Connection header received
0812 14:26:39.351778 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:39.351781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:39.351799 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:39.351889 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:39.352168 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.353357 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.354500 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.355004 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.355095 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.355160 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:39.355317 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 14:26:39.355362 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":131,"threads_started":1}
I20260812 14:26:39.355436 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.351157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:34991 (local address 127.0.0.1:46534)
0812 14:26:39.351642 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:39.351652 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:39.351703 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:39.351970 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:39.351971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:39.351972 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:39.352042 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:39.352044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.353493 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.353494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:39.354392 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:39.354395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.355098 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.355099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:39.355182 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.355196 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 14:26:39.355296 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 14:26:39.355350 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260812 14:26:39.355654 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.351143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:35147 (local address 127.0.0.1:37116)
0812 14:26:39.351326 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:39.351367 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:39.351453 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:39.351954 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:39.351958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:39.351961 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:39.352043 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:39.352044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.353482 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.353483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:39.354552 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:39.354554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.355429 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.355430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:39.355485 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.355488 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:26:39.355517 (+ 29us) client_negotiation.cc:240] Negotiation successful
0812 14:26:39.355568 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":8,"thread_start_us":75,"threads_started":1}
I20260812 14:26:39.356034 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.351263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37116 (local address 127.5.148.2:35147)
0812 14:26:39.351755 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:39.351758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:39.351769 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:26:39.351806 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:39.351808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:39.351825 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:39.352153 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:39.352233 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.353367 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.354641 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.355324 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.355767 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.355842 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:39.355872 (+ 30us) server_negotiation.cc:299] Negotiation successful
0812 14:26:39.355915 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":104,"threads_started":1}
I20260812 14:26:39.356276 28992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.351158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:37787 (local address 127.0.0.1:53160)
0812 14:26:39.351775 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:39.351816 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:39.351858 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:39.352184 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:39.352185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:39.352187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:39.352236 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:39.352237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.353128 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.353129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:39.354396 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:39.354399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.355408 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.355409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:39.355479 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.355484 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:26:39.356003 (+ 519us) client_negotiation.cc:240] Negotiation successful
0812 14:26:39.356055 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":5,"thread_start_us":107,"threads_started":1}
I20260812 14:26:39.356312 28893 tablet_service.cc:1467] Processing CreateTablet for tablet 7ecf7b32c7904cd2ab706ed62bca2da4 (DEFAULT_TABLE table=test-table [id=524b9802f5204e2ba8a2deb9ceb95720]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:39.356524 28995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.351434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53160 (local address 127.5.148.1:37787)
0812 14:26:39.351623 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:39.351628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:39.351930 (+ 302us) server_negotiation.cc:407] Connection header received
0812 14:26:39.351981 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:39.351983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:39.352000 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:39.352072 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:39.352311 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.353022 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.354494 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.355241 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.355313 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.355372 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:39.356029 (+ 657us) server_negotiation.cc:299] Negotiation successful
0812 14:26:39.356442 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":110,"threads_started":1}
I20260812 14:26:39.356731 28893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ecf7b32c7904cd2ab706ed62bca2da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:39.356766 28728 tablet_service.cc:1467] Processing CreateTablet for tablet 7ecf7b32c7904cd2ab706ed62bca2da4 (DEFAULT_TABLE table=test-table [id=524b9802f5204e2ba8a2deb9ceb95720]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:39.357057 28728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ecf7b32c7904cd2ab706ed62bca2da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:39.357210 28573 tablet_service.cc:1467] Processing CreateTablet for tablet 7ecf7b32c7904cd2ab706ed62bca2da4 (DEFAULT_TABLE table=test-table [id=524b9802f5204e2ba8a2deb9ceb95720]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:39.357527 28573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ecf7b32c7904cd2ab706ed62bca2da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:39.359517 28997 tablet_bootstrap.cc:492] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3: Bootstrap starting.
I20260812 14:26:39.360275 28998 tablet_bootstrap.cc:492] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b: Bootstrap starting.
I20260812 14:26:39.360517 28997 tablet_bootstrap.cc:654] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:39.360932 28999 tablet_bootstrap.cc:492] T 7ecf7b32c7904cd2ab706ed62bca2da4 P dbfb6cef20d54318b5b8a4e388996d1e: Bootstrap starting.
I20260812 14:26:39.361135 28998 tablet_bootstrap.cc:654] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:39.361481 28997 log.cc:824] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:39.361693 28999 tablet_bootstrap.cc:654] T 7ecf7b32c7904cd2ab706ed62bca2da4 P dbfb6cef20d54318b5b8a4e388996d1e: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:39.361753 28998 log.cc:824] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:39.362347 28999 log.cc:824] T 7ecf7b32c7904cd2ab706ed62bca2da4 P dbfb6cef20d54318b5b8a4e388996d1e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:39.362469 28997 tablet_bootstrap.cc:492] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3: No bootstrap required, opened a new log
I20260812 14:26:39.362546 28997 ts_tablet_manager.cc:1397] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 14:26:39.362627 28998 tablet_bootstrap.cc:492] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b: No bootstrap required, opened a new log
I20260812 14:26:39.362699 28998 ts_tablet_manager.cc:1397] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:26:39.363051 28999 tablet_bootstrap.cc:492] T 7ecf7b32c7904cd2ab706ed62bca2da4 P dbfb6cef20d54318b5b8a4e388996d1e: No bootstrap required, opened a new log
I20260812 14:26:39.363160 28999 ts_tablet_manager.cc:1397] T 7ecf7b32c7904cd2ab706ed62bca2da4 P dbfb6cef20d54318b5b8a4e388996d1e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:26:39.364670 28998 raft_consensus.cc:348] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92100d9818c44609045097967dfb3b3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35147 } } peers { permanent_uuid: "dbfb6cef20d54318b5b8a4e388996d1e" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 37787 } } peers { permanent_uuid: "073f18bcf5e44c4bae6716000ed3932b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 34991 } }
I20260812 14:26:39.364670 28997 raft_consensus.cc:348] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92100d9818c44609045097967dfb3b3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35147 } } peers { permanent_uuid: "dbfb6cef20d54318b5b8a4e388996d1e" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 37787 } } peers { permanent_uuid: "073f18bcf5e44c4bae6716000ed3932b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 34991 } }
I20260812 14:26:39.364859 28997 raft_consensus.cc:374] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:39.364859 28998 raft_consensus.cc:374] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:39.364899 28997 raft_consensus.cc:729] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92100d9818c44609045097967dfb3b3, State: Initialized, Role: FOLLOWER
I20260812 14:26:39.364909 28998 raft_consensus.cc:729] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 073f18bcf5e44c4bae6716000ed3932b, State: Initialized, Role: FOLLOWER
I20260812 14:26:39.365031 28998 consensus_queue.cc:260] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92100d9818c44609045097967dfb3b3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35147 } } peers { permanent_uuid: "dbfb6cef20d54318b5b8a4e388996d1e" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 37787 } } peers { permanent_uuid: "073f18bcf5e44c4bae6716000ed3932b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 34991 } }
I20260812 14:26:39.365031 28997 consensus_queue.cc:260] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92100d9818c44609045097967dfb3b3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35147 } } peers { permanent_uuid: "dbfb6cef20d54318b5b8a4e388996d1e" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 37787 } } peers { permanent_uuid: "073f18bcf5e44c4bae6716000ed3932b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 34991 } }
I20260812 14:26:39.364806 28999 raft_consensus.cc:348] T 7ecf7b32c7904cd2ab706ed62bca2da4 P dbfb6cef20d54318b5b8a4e388996d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92100d9818c44609045097967dfb3b3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35147 } } peers { permanent_uuid: "dbfb6cef20d54318b5b8a4e388996d1e" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 37787 } } peers { permanent_uuid: "073f18bcf5e44c4bae6716000ed3932b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 34991 } }
I20260812 14:26:39.365239 28999 raft_consensus.cc:374] T 7ecf7b32c7904cd2ab706ed62bca2da4 P dbfb6cef20d54318b5b8a4e388996d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:39.365324 28997 ts_tablet_manager.cc:1428] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 14:26:39.365324 28998 ts_tablet_manager.cc:1428] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:26:39.365327 28999 raft_consensus.cc:729] T 7ecf7b32c7904cd2ab706ed62bca2da4 P dbfb6cef20d54318b5b8a4e388996d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbfb6cef20d54318b5b8a4e388996d1e, State: Initialized, Role: FOLLOWER
I20260812 14:26:39.365454 28977 heartbeater.cc:503] Master 127.5.148.62:39069 was elected leader, sending a full tablet report...
I20260812 14:26:39.365463 28808 heartbeater.cc:503] Master 127.5.148.62:39069 was elected leader, sending a full tablet report...
I20260812 14:26:39.365515 28999 consensus_queue.cc:260] T 7ecf7b32c7904cd2ab706ed62bca2da4 P dbfb6cef20d54318b5b8a4e388996d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92100d9818c44609045097967dfb3b3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35147 } } peers { permanent_uuid: "dbfb6cef20d54318b5b8a4e388996d1e" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 37787 } } peers { permanent_uuid: "073f18bcf5e44c4bae6716000ed3932b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 34991 } }
I20260812 14:26:39.365880 28999 ts_tablet_manager.cc:1428] T 7ecf7b32c7904cd2ab706ed62bca2da4 P dbfb6cef20d54318b5b8a4e388996d1e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:26:39.366066 28653 heartbeater.cc:503] Master 127.5.148.62:39069 was elected leader, sending a full tablet report...
W20260812 14:26:39.377285 28809 tablet.cc:2369] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:26:39.459585 28654 tablet.cc:2369] T 7ecf7b32c7904cd2ab706ed62bca2da4 P dbfb6cef20d54318b5b8a4e388996d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:26:39.574378 28979 tablet.cc:2369] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:26:39.671568 29006 raft_consensus.cc:482] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:26:39.671707 29006 raft_consensus.cc:504] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92100d9818c44609045097967dfb3b3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35147 } } peers { permanent_uuid: "dbfb6cef20d54318b5b8a4e388996d1e" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 37787 } } peers { permanent_uuid: "073f18bcf5e44c4bae6716000ed3932b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 34991 } }
I20260812 14:26:39.672245 29006 leader_election.cc:288] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f92100d9818c44609045097967dfb3b3 (127.5.148.2:35147), dbfb6cef20d54318b5b8a4e388996d1e (127.5.148.1:37787)
I20260812 14:26:39.676224 28995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.672519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:48041 (local address 127.5.148.1:37787)
0812 14:26:39.672573 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:39.672575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:39.672588 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:26:39.672671 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:39.672673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:39.672684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:39.672760 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:39.672992 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.673771 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.675255 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.675807 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.675884 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.675901 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:39.676087 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 14:26:39.676150 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 14:26:39.676223 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.672552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:54247 (local address 127.5.148.2:35147)
0812 14:26:39.672611 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:39.672613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:39.672753 (+ 140us) server_negotiation.cc:407] Connection header received
0812 14:26:39.672830 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:39.672832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:39.672844 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:39.672895 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:39.673075 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.673770 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.675255 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.675807 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.675884 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.675896 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:39.676087 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 14:26:39.676148 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:26:39.676241 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.672458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:37787 (local address 127.5.148.3:48041)
0812 14:26:39.672524 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:39.672537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:39.672585 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:39.672779 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:39.672780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:39.672782 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:39.672895 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:39.672897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.673865 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.673866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:39.675153 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:39.675156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.675901 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.675902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:39.676000 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.676002 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:26:39.676063 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:26:39.676113 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":73}
I20260812 14:26:39.676255 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.672472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:35147 (local address 127.5.148.3:54247)
0812 14:26:39.672658 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:39.672668 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:39.672760 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:39.672918 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:39.672919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:39.672921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:39.672997 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:39.672998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.673865 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.673866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:39.675153 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:39.675156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.675901 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.675902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:39.675998 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.676002 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:26:39.676063 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:26:39.676106 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":44,"thread_start_us":111,"threads_started":1}
I20260812 14:26:39.676640 28598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf7b32c7904cd2ab706ed62bca2da4" candidate_uuid: "073f18bcf5e44c4bae6716000ed3932b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbfb6cef20d54318b5b8a4e388996d1e" is_pre_election: true
I20260812 14:26:39.676803 28598 raft_consensus.cc:2454] T 7ecf7b32c7904cd2ab706ed62bca2da4 P dbfb6cef20d54318b5b8a4e388996d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 073f18bcf5e44c4bae6716000ed3932b in term 0.
I20260812 14:26:39.676928 28750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf7b32c7904cd2ab706ed62bca2da4" candidate_uuid: "073f18bcf5e44c4bae6716000ed3932b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92100d9818c44609045097967dfb3b3" is_pre_election: true
I20260812 14:26:39.677109 28750 raft_consensus.cc:2454] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 073f18bcf5e44c4bae6716000ed3932b in term 0.
I20260812 14:26:39.677093 28836 leader_election.cc:302] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 073f18bcf5e44c4bae6716000ed3932b, dbfb6cef20d54318b5b8a4e388996d1e; no voters:
I20260812 14:26:39.677232 29006 raft_consensus.cc:2781] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:26:39.677304 29006 raft_consensus.cc:482] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:26:39.677331 29006 raft_consensus.cc:3037] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:39.678081 29006 raft_consensus.cc:504] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92100d9818c44609045097967dfb3b3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35147 } } peers { permanent_uuid: "dbfb6cef20d54318b5b8a4e388996d1e" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 37787 } } peers { permanent_uuid: "073f18bcf5e44c4bae6716000ed3932b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 34991 } }
I20260812 14:26:39.678237 29006 leader_election.cc:288] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [CANDIDATE]: Term 1 election: Requested vote from peers f92100d9818c44609045097967dfb3b3 (127.5.148.2:35147), dbfb6cef20d54318b5b8a4e388996d1e (127.5.148.1:37787)
I20260812 14:26:39.678436 28750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf7b32c7904cd2ab706ed62bca2da4" candidate_uuid: "073f18bcf5e44c4bae6716000ed3932b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92100d9818c44609045097967dfb3b3"
I20260812 14:26:39.678494 28598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf7b32c7904cd2ab706ed62bca2da4" candidate_uuid: "073f18bcf5e44c4bae6716000ed3932b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbfb6cef20d54318b5b8a4e388996d1e"
I20260812 14:26:39.678529 28750 raft_consensus.cc:3037] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:39.678565 28598 raft_consensus.cc:3037] T 7ecf7b32c7904cd2ab706ed62bca2da4 P dbfb6cef20d54318b5b8a4e388996d1e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:39.679466 28598 raft_consensus.cc:2454] T 7ecf7b32c7904cd2ab706ed62bca2da4 P dbfb6cef20d54318b5b8a4e388996d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 073f18bcf5e44c4bae6716000ed3932b in term 1.
I20260812 14:26:39.679466 28750 raft_consensus.cc:2454] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 073f18bcf5e44c4bae6716000ed3932b in term 1.
I20260812 14:26:39.679621 28836 leader_election.cc:302] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 073f18bcf5e44c4bae6716000ed3932b, dbfb6cef20d54318b5b8a4e388996d1e; no voters:
I20260812 14:26:39.679723 29006 raft_consensus.cc:2781] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:39.679924 29006 raft_consensus.cc:686] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [term 1 LEADER]: Becoming Leader. State: Replica: 073f18bcf5e44c4bae6716000ed3932b, State: Running, Role: LEADER
I20260812 14:26:39.680034 29006 consensus_queue.cc:237] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92100d9818c44609045097967dfb3b3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35147 } } peers { permanent_uuid: "dbfb6cef20d54318b5b8a4e388996d1e" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 37787 } } peers { permanent_uuid: "073f18bcf5e44c4bae6716000ed3932b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 34991 } }
I20260812 14:26:39.680948 28430 catalog_manager.cc:5495] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b reported cstate change: term changed from 0 to 1, leader changed from <none> to 073f18bcf5e44c4bae6716000ed3932b (127.5.148.3). New cstate: current_term: 1 leader_uuid: "073f18bcf5e44c4bae6716000ed3932b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92100d9818c44609045097967dfb3b3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbfb6cef20d54318b5b8a4e388996d1e" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 37787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "073f18bcf5e44c4bae6716000ed3932b" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 34991 } health_report { overall_health: HEALTHY } } }
I20260812 14:26:39.713361 28996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.710953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.5.148.3:34991)
0812 14:26:39.711030 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:39.711032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:39.711041 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:26:39.711092 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:39.711093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:39.711113 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:39.711177 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:26:39.711319 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.711780 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.712595 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.712788 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.712845 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.712994 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:26:39.713098 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:26:39.713161 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:39.713205 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 14:26:39.713224 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 14:26:39.715245 28750 raft_consensus.cc:1261] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3 [term 1 FOLLOWER]: Refusing update from remote peer 073f18bcf5e44c4bae6716000ed3932b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:26:39.715274 28598 raft_consensus.cc:1261] T 7ecf7b32c7904cd2ab706ed62bca2da4 P dbfb6cef20d54318b5b8a4e388996d1e [term 1 FOLLOWER]: Refusing update from remote peer 073f18bcf5e44c4bae6716000ed3932b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:26:39.715626 29014 consensus_queue.cc:1035] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f92100d9818c44609045097967dfb3b3" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:26:39.715775 29006 consensus_queue.cc:1035] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbfb6cef20d54318b5b8a4e388996d1e" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 37787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:26:39.718999 29018 mvcc.cc:205] Tried to move back new op lower bound from 7317687499630874624 to 7317687499490025472. Current Snapshot: MvccSnapshot[applied={T|T < 7317687499630874624}]
I20260812 14:26:39.720424 28429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:26:39.720522 28429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:26:39.721264 28429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6a71620e0a44735ae593d64ffcc3788: Sending DeleteTablet for 3 replicas of tablet 7ecf7b32c7904cd2ab706ed62bca2da4
I20260812 14:26:39.721573 28728 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ecf7b32c7904cd2ab706ed62bca2da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:39 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:37116
I20260812 14:26:39.721637 28573 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ecf7b32c7904cd2ab706ed62bca2da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:39 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:53160
I20260812 14:26:39.721874 29029 tablet_replica.cc:330] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3: stopping tablet replica
I20260812 14:26:39.721916 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 28498
I20260812 14:26:39.721995 29029 raft_consensus.cc:2229] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:26:39.722110 29029 raft_consensus.cc:2258] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:26:39.722616 29029 ts_tablet_manager.cc:1905] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:26:39.724756 29029 ts_tablet_manager.cc:1918] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:26:39.724867 29029 log.cc:1196] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/7ecf7b32c7904cd2ab706ed62bca2da4
I20260812 14:26:39.725180 29029 ts_tablet_manager.cc:1939] T 7ecf7b32c7904cd2ab706ed62bca2da4 P f92100d9818c44609045097967dfb3b3: Deleting consensus metadata
I20260812 14:26:39.725693 28415 catalog_manager.cc:4841] TS f92100d9818c44609045097967dfb3b3 (127.5.148.2:35147): tablet 7ecf7b32c7904cd2ab706ed62bca2da4 (table test-table [id=524b9802f5204e2ba8a2deb9ceb95720]) successfully deleted
I20260812 14:26:39.728312 28893 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ecf7b32c7904cd2ab706ed62bca2da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:39 UTC) from {username='kudu', principal='kudu/127.5.148.62@KRBTEST.COM'} at 127.0.0.1:46534
I20260812 14:26:39.728739 29033 tablet_replica.cc:330] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b: stopping tablet replica
I20260812 14:26:39.728853 29033 raft_consensus.cc:2229] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:26:39.728971 29033 raft_consensus.cc:2258] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:26:39.729427 29033 ts_tablet_manager.cc:1905] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:26:39.731127 29033 ts_tablet_manager.cc:1918] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:26:39.731202 29033 log.cc:1196] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/7ecf7b32c7904cd2ab706ed62bca2da4
I20260812 14:26:39.731550 29033 ts_tablet_manager.cc:1939] T 7ecf7b32c7904cd2ab706ed62bca2da4 P 073f18bcf5e44c4bae6716000ed3932b: Deleting consensus metadata
I20260812 14:26:39.731977 28414 catalog_manager.cc:4841] TS 073f18bcf5e44c4bae6716000ed3932b (127.5.148.3:34991): tablet 7ecf7b32c7904cd2ab706ed62bca2da4 (table test-table [id=524b9802f5204e2ba8a2deb9ceb95720]) successfully deleted
W20260812 14:26:39.733275 28413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:37787 (error 108)
I20260812 14:26:39.733888 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.733741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:37787 (local address 127.0.0.1:53166)
0812 14:26:39.733805 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:39.733847 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:37787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 14:26:39.734053 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 28658
W20260812 14:26:39.738868 28413 catalog_manager.cc:4584] TS dbfb6cef20d54318b5b8a4e388996d1e (127.5.148.1:37787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ecf7b32c7904cd2ab706ed62bca2da4: Network error: Client connection negotiation failed: client connection to 127.5.148.1:37787: connect: Connection refused (error 111)
I20260812 14:26:39.741832 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 28814
I20260812 14:26:39.753610 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 28394
2026-08-12T14:26:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2388 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T14:26:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:39Z Disabled control of system clock
I20260812 14:26:39.785581 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:40885
--webserver_interface=127.5.148.62
--webserver_port=0
--builtin_ntp_servers=127.5.148.20:40319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:40885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:26:39.868959 29038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:39.869187 29038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:39.869210 29038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:39.870304 29038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:39.870339 29038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:39.870352 29038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:39.870364 29038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:39.870378 29038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:39.871680 29038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:40885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:40885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:39.871878 29038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:39.872170 29038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:39.877036 29046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:39.877029 29044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:39.877188 29038 server_base.cc:1034] running on GCE node
W20260812 14:26:39.877032 29049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:39.877532 29038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:39.877830 29038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:39.879021 29038 hybrid_clock.cc:648] HybridClock initialized: now 1786544799879019 us; error 43 us; skew 500 ppm
I20260812 14:26:39.880414 29038 webserver.cc:460] Webserver started at http://127.5.148.62:44821/ using document root <none> and password file <none>
I20260812 14:26:39.880667 29038 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:39.880740 29038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:39.880894 29038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:39.882020 29038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "b1b7e832c73c45c2b42cd476acbe6540"
format_stamp: "Formatted at 2026-08-12 14:26:39 on dist-test-slave-8w9v"
server_key: "a8e2916df86f4b8a3ce890d01a0c7ede"
server_key_iv: "b44cc7eaff111e291f815575f4ac7ef1"
server_key_version: "encryptionkey@0"
I20260812 14:26:39.882397 29038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "b1b7e832c73c45c2b42cd476acbe6540"
format_stamp: "Formatted at 2026-08-12 14:26:39 on dist-test-slave-8w9v"
server_key: "a8e2916df86f4b8a3ce890d01a0c7ede"
server_key_iv: "b44cc7eaff111e291f815575f4ac7ef1"
server_key_version: "encryptionkey@0"
I20260812 14:26:39.883714 29038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:26:39.884495 29055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:39.884670 29038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:39.884779 29038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "b1b7e832c73c45c2b42cd476acbe6540"
format_stamp: "Formatted at 2026-08-12 14:26:39 on dist-test-slave-8w9v"
server_key: "a8e2916df86f4b8a3ce890d01a0c7ede"
server_key_iv: "b44cc7eaff111e291f815575f4ac7ef1"
server_key_version: "encryptionkey@0"
I20260812 14:26:39.884855 29038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:39.896656 29038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:39.896953 29038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:39.897101 29038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:39.901193 29038 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:40885
I20260812 14:26:39.901237 29120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:40885 every 8 connection(s)
I20260812 14:26:39.901592 29038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:39.902622 29121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:39.904996 29121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540: Bootstrap starting.
I20260812 14:26:39.905552 29121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:39.905799 29121 log.cc:824] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:39.906478 29121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540: No bootstrap required, opened a new log
I20260812 14:26:39.908049 29121 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b7e832c73c45c2b42cd476acbe6540" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40885 } }
I20260812 14:26:39.908262 29121 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:39.908290 29121 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b7e832c73c45c2b42cd476acbe6540, State: Initialized, Role: FOLLOWER
I20260812 14:26:39.908425 29121 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b7e832c73c45c2b42cd476acbe6540" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40885 } }
I20260812 14:26:39.908505 29121 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:39.908528 29121 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:39.908556 29121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:39.909037 29121 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b7e832c73c45c2b42cd476acbe6540" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40885 } }
I20260812 14:26:39.909124 29121 leader_election.cc:302] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1b7e832c73c45c2b42cd476acbe6540; no voters:
I20260812 14:26:39.909247 29121 leader_election.cc:288] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:39.909329 29124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:39.909492 29124 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [term 1 LEADER]: Becoming Leader. State: Replica: b1b7e832c73c45c2b42cd476acbe6540, State: Running, Role: LEADER
I20260812 14:26:39.909540 29121 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:39.909601 29124 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b7e832c73c45c2b42cd476acbe6540" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40885 } }
I20260812 14:26:39.910005 29126 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1b7e832c73c45c2b42cd476acbe6540. Latest consensus state: current_term: 1 leader_uuid: "b1b7e832c73c45c2b42cd476acbe6540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b7e832c73c45c2b42cd476acbe6540" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40885 } } }
I20260812 14:26:39.910091 29125 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1b7e832c73c45c2b42cd476acbe6540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b7e832c73c45c2b42cd476acbe6540" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40885 } } }
I20260812 14:26:39.910107 29126 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:39.910151 29125 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:39.910228 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 29038
I20260812 14:26:39.910326 29129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:39.910346 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:39.910558 5712 external_mini_cluster.cc:1428] Setting key 82c8bb47d24561a016c2bafa302654f4
I20260812 14:26:39.910869 29129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:39.912565 29129 catalog_manager.cc:1347] Generated new cluster ID: fb723da099454fa7aada95fef2fddb70
I20260812 14:26:39.912619 29129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:39.916026 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:39.911194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40584 (local address 127.5.148.62:40885)
0812 14:26:39.911400 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:39.911409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:39.911447 (+ 38us) server_negotiation.cc:407] Connection header received
0812 14:26:39.911582 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:39.911590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:39.911670 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:39.912069 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:39.912190 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.913260 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.914109 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:39.914518 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:39.914650 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:39.915386 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:39.915390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:39.915390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:39.915560 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:39.915593 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:39.915603 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:39.915679 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:39.915757 (+ 78us) server_negotiation.cc:299] Negotiation successful
0812 14:26:39.915840 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260812 14:26:39.924053 29129 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:39.924628 29129 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:39.928575 29129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540: Generated new TSK 0
I20260812 14:26:39.928757 29129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:39.937937 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.1:0
--local_ip_for_outbound_sockets=127.5.148.1
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:40885
--builtin_ntp_servers=127.5.148.20:40319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:26:40.023063 29148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:40.023284 29148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:40.023316 29148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:40.024560 29148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:40.024603 29148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:40.024629 29148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.1
I20260812 14:26:40.026048 29148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.148.1
--webserver_port=0
--tserver_master_addrs=127.5.148.62:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.1
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:40.026337 29148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:40.026647 29148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:40.032696 29154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:40.032716 29153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:40.032820 29156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:40.033074 29148 server_base.cc:1034] running on GCE node
I20260812 14:26:40.033300 29148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:40.033556 29148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:40.034721 29148 hybrid_clock.cc:648] HybridClock initialized: now 1786544800034701 us; error 48 us; skew 500 ppm
I20260812 14:26:40.036183 29148 webserver.cc:460] Webserver started at http://127.5.148.1:33239/ using document root <none> and password file <none>
I20260812 14:26:40.036439 29148 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:40.036511 29148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:40.036652 29148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:40.037815 29148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/data/instance:
uuid: "6350fbbf42a543eeb61773fb76bb3365"
format_stamp: "Formatted at 2026-08-12 14:26:40 on dist-test-slave-8w9v"
server_key: "d565b67cd2240c82be6341ebee82b01a"
server_key_iv: "2c2cd9035024b099bf8bc6c2b2882093"
server_key_version: "encryptionkey@0"
I20260812 14:26:40.038216 29148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance:
uuid: "6350fbbf42a543eeb61773fb76bb3365"
format_stamp: "Formatted at 2026-08-12 14:26:40 on dist-test-slave-8w9v"
server_key: "d565b67cd2240c82be6341ebee82b01a"
server_key_iv: "2c2cd9035024b099bf8bc6c2b2882093"
server_key_version: "encryptionkey@0"
I20260812 14:26:40.039621 29148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:26:40.040474 29162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:40.040704 29148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:40.040808 29148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/wal
uuid: "6350fbbf42a543eeb61773fb76bb3365"
format_stamp: "Formatted at 2026-08-12 14:26:40 on dist-test-slave-8w9v"
server_key: "d565b67cd2240c82be6341ebee82b01a"
server_key_iv: "2c2cd9035024b099bf8bc6c2b2882093"
server_key_version: "encryptionkey@0"
I20260812 14:26:40.040879 29148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:40.072036 29148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:40.072441 29148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:40.072562 29148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:40.072782 29148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:40.073098 29148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:40.073132 29148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:40.073155 29148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:40.073168 29148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:40.079210 29148 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.1:46641
I20260812 14:26:40.079299 29305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.1:46641 every 8 connection(s)
I20260812 14:26:40.079663 29148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/data/info.pb
I20260812 14:26:40.084481 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 29148
I20260812 14:26:40.084614 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-0/wal/instance
I20260812 14:26:40.084897 5712 external_mini_cluster.cc:1428] Setting key ff4f9c56f80e26a894496bc1c4a89a30
I20260812 14:26:40.085134 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.080479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:38431 (local address 127.5.148.62:40885)
0812 14:26:40.080581 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:40.080584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:40.080829 (+ 245us) server_negotiation.cc:407] Connection header received
0812 14:26:40.081322 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:40.081323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:40.081337 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:40.081391 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:40.082082 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.082575 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.083553 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.083738 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.083789 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.084768 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:40.084770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:40.084771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:40.084777 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:40.084798 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:40.084804 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:40.084866 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:40.085009 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 14:26:40.085044 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 14:26:40.085358 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.080447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:40885 (local address 127.5.148.1:38431)
0812 14:26:40.080695 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:40.080738 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:40.081234 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:40.081448 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:40.081452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:40.081460 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:40.082002 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.082007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.082659 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.082659 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.083461 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.083464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.083813 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.083814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.083937 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.084590 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:26:40.084591 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:40.084649 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:26:40.084679 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:40.084883 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:40.084888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:40.084890 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:26:40.084955 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 14:26:40.085085 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":119,"threads_started":1}
I20260812 14:26:40.085911 29306 heartbeater.cc:347] Connected to a master server at 127.5.148.62:40885
I20260812 14:26:40.086015 29306 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:40.086069 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.2:0
--local_ip_for_outbound_sockets=127.5.148.2
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:40885
--builtin_ntp_servers=127.5.148.20:40319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 14:26:40.086232 29306 heartbeater.cc:511] Master 127.5.148.62:40885 requested a full tablet report, sending...
I20260812 14:26:40.086747 29076 ts_manager.cc:194] Registered new tserver with Master: 6350fbbf42a543eeb61773fb76bb3365 (127.5.148.1:46641)
I20260812 14:26:40.087422 29076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.148.1:38431
W20260812 14:26:40.171293 29309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:40.171505 29309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:40.171527 29309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:40.172734 29309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:40.172778 29309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:40.172804 29309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.2
I20260812 14:26:40.174098 29309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.148.2
--webserver_port=0
--tserver_master_addrs=127.5.148.62:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.2
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:40.174283 29309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:40.174533 29309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:40.179618 29316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:40.179633 29320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:40.179606 29315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:40.179849 29309 server_base.cc:1034] running on GCE node
I20260812 14:26:40.180089 29309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:40.180361 29309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:40.181527 29309 hybrid_clock.cc:648] HybridClock initialized: now 1786544800181521 us; error 39 us; skew 500 ppm
I20260812 14:26:40.182813 29309 webserver.cc:460] Webserver started at http://127.5.148.2:44541/ using document root <none> and password file <none>
I20260812 14:26:40.183084 29309 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:40.183157 29309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:40.183321 29309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:40.184516 29309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/data/instance:
uuid: "30c9b83f030c4989ac79ddabc1636c5d"
format_stamp: "Formatted at 2026-08-12 14:26:40 on dist-test-slave-8w9v"
server_key: "233a3edff50eb7fe2cd31fe9d382dffb"
server_key_iv: "72ce04b616b7b24ed34c7b65f6b12037"
server_key_version: "encryptionkey@0"
I20260812 14:26:40.184895 29309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance:
uuid: "30c9b83f030c4989ac79ddabc1636c5d"
format_stamp: "Formatted at 2026-08-12 14:26:40 on dist-test-slave-8w9v"
server_key: "233a3edff50eb7fe2cd31fe9d382dffb"
server_key_iv: "72ce04b616b7b24ed34c7b65f6b12037"
server_key_version: "encryptionkey@0"
I20260812 14:26:40.186268 29309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:26:40.187062 29329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:40.187258 29309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:40.187374 29309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/wal
uuid: "30c9b83f030c4989ac79ddabc1636c5d"
format_stamp: "Formatted at 2026-08-12 14:26:40 on dist-test-slave-8w9v"
server_key: "233a3edff50eb7fe2cd31fe9d382dffb"
server_key_iv: "72ce04b616b7b24ed34c7b65f6b12037"
server_key_version: "encryptionkey@0"
I20260812 14:26:40.187466 29309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:40.201576 29309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:40.201925 29309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:40.202090 29309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:40.202349 29309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:40.202715 29309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:40.202772 29309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:40.202819 29309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:40.202844 29309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:40.208978 29309 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.2:35403
I20260812 14:26:40.209060 29460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.2:35403 every 8 connection(s)
I20260812 14:26:40.209419 29309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/data/info.pb
I20260812 14:26:40.212338 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 29309
I20260812 14:26:40.212464 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/wal/instance
I20260812 14:26:40.212675 5712 external_mini_cluster.cc:1428] Setting key 091014f5df249dd406f935c3f9a8f5d1
I20260812 14:26:40.213641 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.148.3:0
--local_ip_for_outbound_sockets=127.5.148.3
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:40885
--builtin_ntp_servers=127.5.148.20:40319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 14:26:40.215610 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.210297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.2:54763 (local address 127.5.148.62:40885)
0812 14:26:40.210374 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:40.210376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:40.210630 (+ 254us) server_negotiation.cc:407] Connection header received
0812 14:26:40.211271 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:40.211272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:40.211282 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:40.211337 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:40.211996 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.212813 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.214063 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.214273 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.214358 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.215322 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:40.215324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:40.215324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:40.215330 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:40.215351 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:40.215357 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:40.215419 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:40.215524 (+ 105us) server_negotiation.cc:299] Negotiation successful
0812 14:26:40.215549 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:26:40.215905 29463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.210264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:40885 (local address 127.5.148.2:54763)
0812 14:26:40.210486 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:40.210522 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:40.211163 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:40.211393 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:40.211396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:40.211405 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:40.211911 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.211926 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.212925 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.212927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.213973 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.213976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.214371 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.214372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.214521 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.215118 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:26:40.215118 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:40.215176 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:26:40.215208 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:40.215435 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:40.215442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:40.215446 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:26:40.215520 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 14:26:40.215639 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":106,"threads_started":1}
I20260812 14:26:40.216459 29461 heartbeater.cc:347] Connected to a master server at 127.5.148.62:40885
I20260812 14:26:40.216578 29461 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:40.216785 29461 heartbeater.cc:511] Master 127.5.148.62:40885 requested a full tablet report, sending...
I20260812 14:26:40.217162 29076 ts_manager.cc:194] Registered new tserver with Master: 30c9b83f030c4989ac79ddabc1636c5d (127.5.148.2:35403)
I20260812 14:26:40.217540 29076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.148.2:54763
W20260812 14:26:40.298694 29464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:40.298907 29464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:40.298929 29464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:40.300091 29464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:40.300186 29464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:40.300242 29464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.148.3
I20260812 14:26:40.301647 29464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:40319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.148.3
--webserver_port=0
--tserver_master_addrs=127.5.148.62:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.148.3
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:40.301908 29464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:40.302222 29464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:40.307544 29472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:40.307555 29470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:40.307586 29464 server_base.cc:1034] running on GCE node
W20260812 14:26:40.307551 29469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:40.307974 29464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:40.308336 29464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:40.309504 29464 hybrid_clock.cc:648] HybridClock initialized: now 1786544800309498 us; error 36 us; skew 500 ppm
I20260812 14:26:40.310863 29464 webserver.cc:460] Webserver started at http://127.5.148.3:35469/ using document root <none> and password file <none>
I20260812 14:26:40.311203 29464 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:40.311270 29464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:40.311442 29464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:40.312808 29464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/data/instance:
uuid: "28f69e46057541cd9fc17cdbbda97b62"
format_stamp: "Formatted at 2026-08-12 14:26:40 on dist-test-slave-8w9v"
server_key: "be70494bdb7286ff72c6e69b49835ff3"
server_key_iv: "bc4209aa2ccc19d607f48b00709e39a3"
server_key_version: "encryptionkey@0"
I20260812 14:26:40.313195 29464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance:
uuid: "28f69e46057541cd9fc17cdbbda97b62"
format_stamp: "Formatted at 2026-08-12 14:26:40 on dist-test-slave-8w9v"
server_key: "be70494bdb7286ff72c6e69b49835ff3"
server_key_iv: "bc4209aa2ccc19d607f48b00709e39a3"
server_key_version: "encryptionkey@0"
I20260812 14:26:40.314559 29464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:26:40.315284 29479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:40.315454 29464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:26:40.315557 29464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/wal
uuid: "28f69e46057541cd9fc17cdbbda97b62"
format_stamp: "Formatted at 2026-08-12 14:26:40 on dist-test-slave-8w9v"
server_key: "be70494bdb7286ff72c6e69b49835ff3"
server_key_iv: "bc4209aa2ccc19d607f48b00709e39a3"
server_key_version: "encryptionkey@0"
I20260812 14:26:40.315641 29464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:40.340740 29464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:40.341086 29464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:40.341248 29464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:40.341495 29464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:26:40.341859 29464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:26:40.341926 29464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:40.341992 29464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:26:40.342024 29464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:40.348567 29464 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.3:46817
I20260812 14:26:40.348614 29617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.3:46817 every 8 connection(s)
I20260812 14:26:40.348961 29464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/data/info.pb
I20260812 14:26:40.350479 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 29464
I20260812 14:26:40.350621 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/wal/instance
I20260812 14:26:40.350843 5712 external_mini_cluster.cc:1428] Setting key 945a6361f158acd558ecccb163a975d9
I20260812 14:26:40.354588 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.349712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.3:51985 (local address 127.5.148.62:40885)
0812 14:26:40.349779 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:40.349781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:40.350026 (+ 245us) server_negotiation.cc:407] Connection header received
0812 14:26:40.350610 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:40.350611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:40.350625 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:40.350683 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:40.351595 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.352183 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.353170 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.353371 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.353448 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.354270 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:40.354271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:40.354272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:40.354276 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:40.354292 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:40.354297 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:40.354366 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:40.354476 (+ 110us) server_negotiation.cc:299] Negotiation successful
0812 14:26:40.354508 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:26:40.354812 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.349699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:40885 (local address 127.5.148.3:51985)
0812 14:26:40.349892 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:40.349932 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:40.350499 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:40.350762 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:40.350766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:40.350774 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:26:40.351515 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.351522 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.352283 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.352284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.353074 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.353076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.353459 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.353461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.353607 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.354125 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:26:40.354125 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:26:40.354154 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:26:40.354178 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:26:40.354368 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:26:40.354375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:26:40.354378 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:26:40.354479 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 14:26:40.354581 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260812 14:26:40.355302 29619 heartbeater.cc:347] Connected to a master server at 127.5.148.62:40885
I20260812 14:26:40.355412 29619 heartbeater.cc:464] Registering TS with master...
I20260812 14:26:40.355571 29619 heartbeater.cc:511] Master 127.5.148.62:40885 requested a full tablet report, sending...
I20260812 14:26:40.355938 29076 ts_manager.cc:194] Registered new tserver with Master: 28f69e46057541cd9fc17cdbbda97b62 (127.5.148.3:46817)
I20260812 14:26:40.356326 29076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.148.3:51985
I20260812 14:26:40.363019 5712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:26:40.367656 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.364563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40586 (local address 127.5.148.62:40885)
0812 14:26:40.364654 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:40.364656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:40.364771 (+ 115us) server_negotiation.cc:407] Connection header received
0812 14:26:40.364873 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:40.364874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:40.364896 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:40.364966 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:40.365159 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.365653 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.366422 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.366623 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.366702 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.367370 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:40.367371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:40.367372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:40.367377 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:40.367392 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:40.367396 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:40.367468 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:40.367532 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 14:26:40.367563 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:26:40.369277 29076 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:26:40.370910 29076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:26:40.377935 29633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.373526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.5.148.1:46641)
0812 14:26:40.373735 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:40.373739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:40.373749 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:26:40.373789 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:40.373791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:40.373805 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:40.373873 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:40.374080 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.375335 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.376973 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.377497 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.377599 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.377703 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:40.377790 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 14:26:40.377828 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":126,"threads_started":1}
I20260812 14:26:40.378012 29629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.373123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:35403 (local address 127.0.0.1:56592)
0812 14:26:40.373313 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:40.373344 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:40.373413 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:40.373853 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:40.373857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:40.373859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:40.373933 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.373935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.375460 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.375461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.376685 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.376688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.377641 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.377643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.377710 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.377716 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 14:26:40.377750 (+ 34us) client_negotiation.cc:240] Negotiation successful
0812 14:26:40.377824 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":107,"thread_start_us":113,"threads_started":1}
I20260812 14:26:40.378155 29631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.373247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56592 (local address 127.5.148.2:35403)
0812 14:26:40.373677 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:40.373682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:40.373694 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:26:40.373739 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:40.373741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:40.373760 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:40.373843 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:40.374033 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.375335 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.376777 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.377529 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.377879 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.377993 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:40.378021 (+ 28us) server_negotiation.cc:299] Negotiation successful
0812 14:26:40.378066 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":94,"threads_started":1}
I20260812 14:26:40.377978 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.373349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:46817 (local address 127.0.0.1:41904)
0812 14:26:40.373492 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:40.373502 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:40.373547 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:40.374193 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:40.374194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:40.374196 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:40.374228 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.374230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.375456 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.375457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.376855 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.376858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.377769 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.377769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.377821 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.377825 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:26:40.377855 (+ 30us) client_negotiation.cc:240] Negotiation successful
0812 14:26:40.377887 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":109,"thread_start_us":93,"threads_started":1}
I20260812 14:26:40.377969 29630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.373089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:46641 (local address 127.0.0.1:33574)
0812 14:26:40.373300 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:40.373343 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:40.373412 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:40.373936 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:40.373937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:40.373939 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:40.373992 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.373994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.375428 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.375429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.376855 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.376858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.377599 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.377601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.377676 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.377690 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 14:26:40.377769 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 14:26:40.377822 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":58,"thread_start_us":91,"threads_started":1}
I20260812 14:26:40.378623 29634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.373651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.5.148.3:46817)
0812 14:26:40.374044 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:40.374048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:40.374058 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:26:40.374100 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:40.374102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:40.374118 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:40.374198 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:40.374293 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.375362 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.376950 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.377676 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.378346 (+ 670us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.378463 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:40.378490 (+ 27us) server_negotiation.cc:299] Negotiation successful
0812 14:26:40.378533 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":90,"threads_started":1}
I20260812 14:26:40.378932 29384 tablet_service.cc:1467] Processing CreateTablet for tablet 4281e6148c154ba1b213b1865db0c012 (DEFAULT_TABLE table=test-table [id=8443c315984246cfaa96c504288b2ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:40.379340 29384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4281e6148c154ba1b213b1865db0c012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:40.379269 29537 tablet_service.cc:1467] Processing CreateTablet for tablet 4281e6148c154ba1b213b1865db0c012 (DEFAULT_TABLE table=test-table [id=8443c315984246cfaa96c504288b2ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:40.379623 29537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4281e6148c154ba1b213b1865db0c012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:40.383065 29636 tablet_bootstrap.cc:492] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d: Bootstrap starting.
I20260812 14:26:40.383834 29636 tablet_bootstrap.cc:654] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:40.383798 29228 tablet_service.cc:1467] Processing CreateTablet for tablet 4281e6148c154ba1b213b1865db0c012 (DEFAULT_TABLE table=test-table [id=8443c315984246cfaa96c504288b2ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:26:40.384254 29228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4281e6148c154ba1b213b1865db0c012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:40.384250 29636 log.cc:824] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:40.384318 29638 tablet_bootstrap.cc:492] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62: Bootstrap starting.
I20260812 14:26:40.385035 29638 tablet_bootstrap.cc:654] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:40.385313 29636 tablet_bootstrap.cc:492] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d: No bootstrap required, opened a new log
I20260812 14:26:40.385396 29636 ts_tablet_manager.cc:1397] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:26:40.385406 29638 log.cc:824] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:40.386093 29638 tablet_bootstrap.cc:492] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62: No bootstrap required, opened a new log
I20260812 14:26:40.386178 29638 ts_tablet_manager.cc:1397] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:26:40.388089 29642 tablet_bootstrap.cc:492] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365: Bootstrap starting.
I20260812 14:26:40.388197 29636 raft_consensus.cc:348] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6350fbbf42a543eeb61773fb76bb3365" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46641 } } peers { permanent_uuid: "30c9b83f030c4989ac79ddabc1636c5d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35403 } } peers { permanent_uuid: "28f69e46057541cd9fc17cdbbda97b62" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46817 } }
I20260812 14:26:40.388414 29636 raft_consensus.cc:374] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:40.388468 29636 raft_consensus.cc:729] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c9b83f030c4989ac79ddabc1636c5d, State: Initialized, Role: FOLLOWER
I20260812 14:26:40.388407 29638 raft_consensus.cc:348] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6350fbbf42a543eeb61773fb76bb3365" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46641 } } peers { permanent_uuid: "30c9b83f030c4989ac79ddabc1636c5d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35403 } } peers { permanent_uuid: "28f69e46057541cd9fc17cdbbda97b62" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46817 } }
I20260812 14:26:40.388581 29638 raft_consensus.cc:374] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:40.388628 29638 raft_consensus.cc:729] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f69e46057541cd9fc17cdbbda97b62, State: Initialized, Role: FOLLOWER
I20260812 14:26:40.388616 29636 consensus_queue.cc:260] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6350fbbf42a543eeb61773fb76bb3365" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46641 } } peers { permanent_uuid: "30c9b83f030c4989ac79ddabc1636c5d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35403 } } peers { permanent_uuid: "28f69e46057541cd9fc17cdbbda97b62" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46817 } }
I20260812 14:26:40.388768 29638 consensus_queue.cc:260] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6350fbbf42a543eeb61773fb76bb3365" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46641 } } peers { permanent_uuid: "30c9b83f030c4989ac79ddabc1636c5d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35403 } } peers { permanent_uuid: "28f69e46057541cd9fc17cdbbda97b62" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46817 } }
I20260812 14:26:40.388929 29636 ts_tablet_manager.cc:1428] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 14:26:40.389046 29461 heartbeater.cc:503] Master 127.5.148.62:40885 was elected leader, sending a full tablet report...
I20260812 14:26:40.389067 29638 ts_tablet_manager.cc:1428] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:26:40.389185 29619 heartbeater.cc:503] Master 127.5.148.62:40885 was elected leader, sending a full tablet report...
I20260812 14:26:40.389305 29642 tablet_bootstrap.cc:654] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:40.389645 29642 log.cc:824] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:40.390515 29642 tablet_bootstrap.cc:492] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365: No bootstrap required, opened a new log
I20260812 14:26:40.390573 29642 ts_tablet_manager.cc:1397] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 14:26:40.392712 29642 raft_consensus.cc:348] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6350fbbf42a543eeb61773fb76bb3365" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46641 } } peers { permanent_uuid: "30c9b83f030c4989ac79ddabc1636c5d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35403 } } peers { permanent_uuid: "28f69e46057541cd9fc17cdbbda97b62" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46817 } }
I20260812 14:26:40.392853 29642 raft_consensus.cc:374] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:40.392876 29642 raft_consensus.cc:729] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6350fbbf42a543eeb61773fb76bb3365, State: Initialized, Role: FOLLOWER
I20260812 14:26:40.393054 29642 consensus_queue.cc:260] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6350fbbf42a543eeb61773fb76bb3365" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46641 } } peers { permanent_uuid: "30c9b83f030c4989ac79ddabc1636c5d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35403 } } peers { permanent_uuid: "28f69e46057541cd9fc17cdbbda97b62" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46817 } }
I20260812 14:26:40.393321 29642 ts_tablet_manager.cc:1428] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 14:26:40.393402 29306 heartbeater.cc:503] Master 127.5.148.62:40885 was elected leader, sending a full tablet report...
W20260812 14:26:40.460111 29462 tablet.cc:2369] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:26:40.580479 29307 tablet.cc:2369] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:26:40.599644 29622 tablet.cc:2369] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:26:40.641230 29647 raft_consensus.cc:482] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:26:40.641373 29647 raft_consensus.cc:504] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6350fbbf42a543eeb61773fb76bb3365" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46641 } } peers { permanent_uuid: "30c9b83f030c4989ac79ddabc1636c5d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35403 } } peers { permanent_uuid: "28f69e46057541cd9fc17cdbbda97b62" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46817 } }
I20260812 14:26:40.641799 29647 leader_election.cc:288] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30c9b83f030c4989ac79ddabc1636c5d (127.5.148.2:35403), 28f69e46057541cd9fc17cdbbda97b62 (127.5.148.3:46817)
I20260812 14:26:40.646013 29634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.642149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:54327 (local address 127.5.148.3:46817)
0812 14:26:40.642195 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:40.642197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:40.642281 (+ 84us) server_negotiation.cc:407] Connection header received
0812 14:26:40.642331 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:40.642332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:40.642343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:40.642411 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:40.642644 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.643392 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.645016 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.645593 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.645673 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.645683 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:40.645900 (+ 217us) server_negotiation.cc:299] Negotiation successful
0812 14:26:40.645945 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:26:40.646013 29631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.642105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.148.1:37689 (local address 127.5.148.2:35403)
0812 14:26:40.642163 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:40.642166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:40.642282 (+ 116us) server_negotiation.cc:407] Connection header received
0812 14:26:40.642326 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:40.642327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:40.642340 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:40.642411 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:26:40.642644 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.643392 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.645015 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.645593 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.645673 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.645683 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:40.645887 (+ 204us) server_negotiation.cc:299] Negotiation successful
0812 14:26:40.645940 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:26:40.646003 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.642011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.2:35403 (local address 127.5.148.1:37689)
0812 14:26:40.642206 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:40.642215 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:40.642269 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:40.642448 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:40.642449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:40.642451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:40.642559 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.642562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.643492 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.643493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.644917 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.644921 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.645696 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.645696 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.645763 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.645770 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 14:26:40.645836 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 14:26:40.645899 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":89,"spinlock_wait_cycles":14720,"thread_start_us":68,"threads_started":1}
I20260812 14:26:40.646003 29648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.642011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.3:46817 (local address 127.5.148.1:54327)
0812 14:26:40.642174 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:40.642187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:26:40.642269 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:26:40.642448 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:26:40.642449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:26:40.642452 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:26:40.642559 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.642562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.643492 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.643493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.644917 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:26:40.644921 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.645684 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.645685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:26:40.645763 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.645772 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 14:26:40.645836 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 14:26:40.645887 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":115,"thread_start_us":83,"threads_started":1}
I20260812 14:26:40.646618 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4281e6148c154ba1b213b1865db0c012" candidate_uuid: "6350fbbf42a543eeb61773fb76bb3365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f69e46057541cd9fc17cdbbda97b62" is_pre_election: true
I20260812 14:26:40.646620 29409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4281e6148c154ba1b213b1865db0c012" candidate_uuid: "6350fbbf42a543eeb61773fb76bb3365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c9b83f030c4989ac79ddabc1636c5d" is_pre_election: true
I20260812 14:26:40.646788 29561 raft_consensus.cc:2454] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6350fbbf42a543eeb61773fb76bb3365 in term 0.
I20260812 14:26:40.646788 29409 raft_consensus.cc:2454] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6350fbbf42a543eeb61773fb76bb3365 in term 0.
I20260812 14:26:40.647058 29166 leader_election.cc:302] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c9b83f030c4989ac79ddabc1636c5d, 6350fbbf42a543eeb61773fb76bb3365; no voters:
I20260812 14:26:40.647212 29647 raft_consensus.cc:2781] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:26:40.647287 29647 raft_consensus.cc:482] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:26:40.647360 29647 raft_consensus.cc:3037] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:40.648245 29647 raft_consensus.cc:504] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6350fbbf42a543eeb61773fb76bb3365" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46641 } } peers { permanent_uuid: "30c9b83f030c4989ac79ddabc1636c5d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35403 } } peers { permanent_uuid: "28f69e46057541cd9fc17cdbbda97b62" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46817 } }
I20260812 14:26:40.648408 29647 leader_election.cc:288] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [CANDIDATE]: Term 1 election: Requested vote from peers 30c9b83f030c4989ac79ddabc1636c5d (127.5.148.2:35403), 28f69e46057541cd9fc17cdbbda97b62 (127.5.148.3:46817)
I20260812 14:26:40.648603 29409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4281e6148c154ba1b213b1865db0c012" candidate_uuid: "6350fbbf42a543eeb61773fb76bb3365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c9b83f030c4989ac79ddabc1636c5d"
I20260812 14:26:40.648633 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4281e6148c154ba1b213b1865db0c012" candidate_uuid: "6350fbbf42a543eeb61773fb76bb3365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f69e46057541cd9fc17cdbbda97b62"
I20260812 14:26:40.648711 29561 raft_consensus.cc:3037] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:40.648701 29409 raft_consensus.cc:3037] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:40.649667 29561 raft_consensus.cc:2454] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6350fbbf42a543eeb61773fb76bb3365 in term 1.
I20260812 14:26:40.649667 29409 raft_consensus.cc:2454] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6350fbbf42a543eeb61773fb76bb3365 in term 1.
I20260812 14:26:40.649935 29167 leader_election.cc:302] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28f69e46057541cd9fc17cdbbda97b62, 6350fbbf42a543eeb61773fb76bb3365; no voters:
I20260812 14:26:40.650074 29647 raft_consensus.cc:2781] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:40.650256 29647 raft_consensus.cc:686] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [term 1 LEADER]: Becoming Leader. State: Replica: 6350fbbf42a543eeb61773fb76bb3365, State: Running, Role: LEADER
I20260812 14:26:40.650367 29647 consensus_queue.cc:237] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6350fbbf42a543eeb61773fb76bb3365" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46641 } } peers { permanent_uuid: "30c9b83f030c4989ac79ddabc1636c5d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35403 } } peers { permanent_uuid: "28f69e46057541cd9fc17cdbbda97b62" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46817 } }
I20260812 14:26:40.651319 29075 catalog_manager.cc:5495] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6350fbbf42a543eeb61773fb76bb3365 (127.5.148.1). New cstate: current_term: 1 leader_uuid: "6350fbbf42a543eeb61773fb76bb3365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6350fbbf42a543eeb61773fb76bb3365" member_type: VOTER last_known_addr { host: "127.5.148.1" port: 46641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c9b83f030c4989ac79ddabc1636c5d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28f69e46057541cd9fc17cdbbda97b62" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20260812 14:26:40.659842 29633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.657727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.5.148.1:46641)
0812 14:26:40.657770 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:40.657772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:40.657780 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:26:40.657822 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:40.657823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:40.657840 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:40.657909 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:26:40.658001 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.658493 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.659203 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:40.659389 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:40.659440 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:40.659575 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:26:40.659674 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:26:40.659736 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:40.659767 (+ 31us) server_negotiation.cc:299] Negotiation successful
0812 14:26:40.659785 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:26:40.661834 29409 raft_consensus.cc:1261] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d [term 1 FOLLOWER]: Refusing update from remote peer 6350fbbf42a543eeb61773fb76bb3365: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:26:40.661855 29561 raft_consensus.cc:1261] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62 [term 1 FOLLOWER]: Refusing update from remote peer 6350fbbf42a543eeb61773fb76bb3365: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:26:40.662253 29647 consensus_queue.cc:1035] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28f69e46057541cd9fc17cdbbda97b62" member_type: VOTER last_known_addr { host: "127.5.148.3" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:26:40.662462 29650 consensus_queue.cc:1035] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c9b83f030c4989ac79ddabc1636c5d" member_type: VOTER last_known_addr { host: "127.5.148.2" port: 35403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:26:40.665759 29655 mvcc.cc:205] Tried to move back new op lower bound from 7317687503507615744 to 7317687503464452096. Current Snapshot: MvccSnapshot[applied={T|T < 7317687503507615744 or (T in {7317687503507615744})}]
I20260812 14:26:40.665889 29656 mvcc.cc:205] Tried to move back new op lower bound from 7317687503507615744 to 7317687503464452096. Current Snapshot: MvccSnapshot[applied={T|T < 7317687503507615744 or (T in {7317687503507615744})}]
I20260812 14:26:40.667465 29076 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:26:40.667595 29076 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:26:40.668421 29076 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1b7e832c73c45c2b42cd476acbe6540: Sending DeleteTablet for 3 replicas of tablet 4281e6148c154ba1b213b1865db0c012
I20260812 14:26:40.668745 29228 tablet_service.cc:1514] Processing DeleteTablet for tablet 4281e6148c154ba1b213b1865db0c012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:40 UTC) from {username='slave'} at 127.0.0.1:33574
I20260812 14:26:40.668792 29537 tablet_service.cc:1514] Processing DeleteTablet for tablet 4281e6148c154ba1b213b1865db0c012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:40 UTC) from {username='slave'} at 127.0.0.1:41904
I20260812 14:26:40.669019 29667 tablet_replica.cc:330] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62: stopping tablet replica
I20260812 14:26:40.669121 29667 raft_consensus.cc:2229] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:26:40.669242 29667 raft_consensus.cc:2258] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:26:40.669315 29384 tablet_service.cc:1514] Processing DeleteTablet for tablet 4281e6148c154ba1b213b1865db0c012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:26:40 UTC) from {username='slave'} at 127.0.0.1:56592
I20260812 14:26:40.669413 29666 tablet_replica.cc:330] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365: stopping tablet replica
I20260812 14:26:40.669529 29666 raft_consensus.cc:2229] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:26:40.669620 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 29148
I20260812 14:26:40.669639 29666 raft_consensus.cc:2258] T 4281e6148c154ba1b213b1865db0c012 P 6350fbbf42a543eeb61773fb76bb3365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:26:40.669633 29668 tablet_replica.cc:330] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d: stopping tablet replica
I20260812 14:26:40.669842 29668 raft_consensus.cc:2229] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:26:40.670006 29668 raft_consensus.cc:2258] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:26:40.670536 29668 ts_tablet_manager.cc:1905] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:26:40.672760 29668 ts_tablet_manager.cc:1918] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:26:40.672899 29668 log.cc:1196] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-1/wal/wals/4281e6148c154ba1b213b1865db0c012
I20260812 14:26:40.673223 29668 ts_tablet_manager.cc:1939] T 4281e6148c154ba1b213b1865db0c012 P 30c9b83f030c4989ac79ddabc1636c5d: Deleting consensus metadata
I20260812 14:26:40.673740 29058 catalog_manager.cc:4841] TS 30c9b83f030c4989ac79ddabc1636c5d (127.5.148.2:35403): tablet 4281e6148c154ba1b213b1865db0c012 (table test-table [id=8443c315984246cfaa96c504288b2ae3]) successfully deleted
I20260812 14:26:40.673977 29667 ts_tablet_manager.cc:1905] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:26:40.675478 29667 ts_tablet_manager.cc:1918] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:26:40.675547 29667 log.cc:1196] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62: Deleting WAL directory at /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786544547303129-5712-0/minicluster-data/ts-2/wal/wals/4281e6148c154ba1b213b1865db0c012
I20260812 14:26:40.675740 29667 ts_tablet_manager.cc:1939] T 4281e6148c154ba1b213b1865db0c012 P 28f69e46057541cd9fc17cdbbda97b62: Deleting consensus metadata
I20260812 14:26:40.676205 29060 catalog_manager.cc:4841] TS 28f69e46057541cd9fc17cdbbda97b62 (127.5.148.3:46817): tablet 4281e6148c154ba1b213b1865db0c012 (table test-table [id=8443c315984246cfaa96c504288b2ae3]) successfully deleted
W20260812 14:26:40.678400 29060 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.148.1:46641 (error 108)
I20260812 14:26:40.678623 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 29309
I20260812 14:26:40.680976 29630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:40.678761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.1:46641 (local address 127.0.0.1:33592)
0812 14:26:40.680884 (+ 2123us) negotiation.cc:107] Waiting for socket to connect
0812 14:26:40.680944 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.1:46641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2096}
W20260812 14:26:40.681128 29060 catalog_manager.cc:4584] TS 6350fbbf42a543eeb61773fb76bb3365 (127.5.148.1:46641): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4281e6148c154ba1b213b1865db0c012: Network error: Client connection negotiation failed: client connection to 127.5.148.1:46641: connect: Connection refused (error 111)
I20260812 14:26:40.688301 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 29464
I20260812 14:26:40.699342 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 29038
2026-08-12T14:26:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (945 ms)
[----------] 6 tests from EncryptionPolicyTest (9042 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:40 dist-test-slave-8w9v krb5kdc[29672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:40 dist-test-slave-8w9v krb5kdc[29672](info): set up 2 sockets
Aug 12 14:26:40 dist-test-slave-8w9v krb5kdc[29672](info): commencing operation
krb5kdc: starting...
W20260812 14:26:41.825551 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:41 dist-test-slave-8w9v krb5kdc[29672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:26:41.952862 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:42.038769 29686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:42.038990 29686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:42.039081 29686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:42.040386 29686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:42.040437 29686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:42.040455 29686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:42.040470 29686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:42.040534 29686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:42.042088 29686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36967
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:42.042382 29686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:42.042644 29686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:42.047802 29693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:42.047869 29692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:42.047845 29696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:42.048192 29686 server_base.cc:1034] running on GCE node
I20260812 14:26:42.048463 29686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:42.048791 29686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:42.049942 29686 hybrid_clock.cc:648] HybridClock initialized: now 1786544802049924 us; error 38 us; skew 500 ppm
Aug 12 14:26:42 dist-test-slave-8w9v krb5kdc[29672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:42.053059 29686 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:26:42.053550 29686 webserver.cc:460] Webserver started at http://127.0.0.1:43711/ using document root <none> and password file <none>
I20260812 14:26:42.053803 29686 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:42.053850 29686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:42.054041 29686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:42.055119 29686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "e4b2581de37a4633a73cb35aa4877a05"
format_stamp: "Formatted at 2026-08-12 14:26:42 on dist-test-slave-8w9v"
server_key: "85ef13234656e597f5fc6931ad7de581"
server_key_iv: "44bae466483ec4a81ccea43de8690ea4"
server_key_version: "encryptionkey@0"
I20260812 14:26:42.055495 29686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "e4b2581de37a4633a73cb35aa4877a05"
format_stamp: "Formatted at 2026-08-12 14:26:42 on dist-test-slave-8w9v"
server_key: "85ef13234656e597f5fc6931ad7de581"
server_key_iv: "44bae466483ec4a81ccea43de8690ea4"
server_key_version: "encryptionkey@0"
I20260812 14:26:42.056988 29686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:26:42.057874 29705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:42.058109 29686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:42.058221 29686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "e4b2581de37a4633a73cb35aa4877a05"
format_stamp: "Formatted at 2026-08-12 14:26:42 on dist-test-slave-8w9v"
server_key: "85ef13234656e597f5fc6931ad7de581"
server_key_iv: "44bae466483ec4a81ccea43de8690ea4"
server_key_version: "encryptionkey@0"
I20260812 14:26:42.058297 29686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:42.068774 29686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:42.071296 29686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:42.071482 29686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:42.076320 29686 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36967
I20260812 14:26:42.076388 29765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36967 every 8 connection(s)
I20260812 14:26:42.076776 29686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:42.077800 29766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:42.078770 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 29686
I20260812 14:26:42.078897 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:42.079107 5712 external_mini_cluster.cc:1428] Setting key afc539096c7ccfbddfd6431b8757cfab
I20260812 14:26:42.081414 29766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05: Bootstrap starting.
I20260812 14:26:42.082033 29766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:42.082348 29766 log.cc:824] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:42.083105 29766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05: No bootstrap required, opened a new log
I20260812 14:26:42.085757 29766 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b2581de37a4633a73cb35aa4877a05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36967 } }
I20260812 14:26:42.085898 29766 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:42.085928 29766 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b2581de37a4633a73cb35aa4877a05, State: Initialized, Role: FOLLOWER
I20260812 14:26:42.086077 29766 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b2581de37a4633a73cb35aa4877a05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36967 } }
I20260812 14:26:42.086153 29766 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:42.086174 29766 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:42.086241 29766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [term 0 FOLLOWER]: Advancing to term 1
Aug 12 14:26:42 dist-test-slave-8w9v krb5kdc[29672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 14:26:42.087018 29766 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b2581de37a4633a73cb35aa4877a05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36967 } }
I20260812 14:26:42.087145 29766 leader_election.cc:302] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4b2581de37a4633a73cb35aa4877a05; no voters:
I20260812 14:26:42.087355 29766 leader_election.cc:288] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:42.087503 29773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:42.087646 29773 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [term 1 LEADER]: Becoming Leader. State: Replica: e4b2581de37a4633a73cb35aa4877a05, State: Running, Role: LEADER
I20260812 14:26:42.087661 29766 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:42.087749 29773 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b2581de37a4633a73cb35aa4877a05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36967 } }
I20260812 14:26:42.088441 29774 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4b2581de37a4633a73cb35aa4877a05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b2581de37a4633a73cb35aa4877a05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36967 } } }
I20260812 14:26:42.088543 29774 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:42.088862 29775 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4b2581de37a4633a73cb35aa4877a05. Latest consensus state: current_term: 1 leader_uuid: "e4b2581de37a4633a73cb35aa4877a05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b2581de37a4633a73cb35aa4877a05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36967 } } }
I20260812 14:26:42.089015 29775 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05 [sys.catalog]: This master's current role is: LEADER
W20260812 14:26:42.089618 29795 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:26:42.089686 29795 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:26:42.089761 29787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:42.089968 29787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:42.090996 29771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:42.079691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40491 (local address 127.0.0.1:36967)
0812 14:26:42.079872 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:42.079880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:42.079917 (+ 37us) server_negotiation.cc:407] Connection header received
0812 14:26:42.080369 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:42.080376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:42.080423 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:42.080538 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:42.081360 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:42.082372 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:42.083397 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:42.083793 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:42.083884 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:42.086876 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:42.086880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:42.086880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:42.086900 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:42.089175 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:42.089779 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:42.089782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:42.089783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:42.089879 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:42.090131 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:42.090131 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:42.090132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:42.090364 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:42.090483 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:42.090699 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 14:26:42.090806 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260812 14:26:42.091598 29787 catalog_manager.cc:1347] Generated new cluster ID: 2455b819333d475e815235c0ab4452d4
I20260812 14:26:42.091651 29787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:42.092005 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:26:42.101646 29771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:42.093646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59113 (local address 127.0.0.1:36967)
0812 14:26:42.093682 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:42.093683 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:42.093958 (+ 275us) server_negotiation.cc:407] Connection header received
0812 14:26:42.094056 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:42.094057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:42.094068 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:42.094097 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:42.094839 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:42.095593 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:42.096732 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:42.097076 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:42.097120 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:42.098659 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:42.098660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:42.098660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:42.098667 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:42.100256 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:42.100702 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:42.100703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:42.100704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:42.100762 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:42.101025 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:42.101026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:42.101026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:42.101153 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:42.101233 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:42.101485 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 14:26:42.101551 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:26:42.121934 29787 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:42.122709 29787 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:42.128876 29787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4b2581de37a4633a73cb35aa4877a05: Generated new TSK 0
I20260812 14:26:42.129136 29787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:42.134670 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 29686
W20260812 14:26:42.146795 29670 connection.cc:535] client connection to 127.0.0.1:36967 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36967): Connection reset by peer (error 104)
2026-08-12T14:26:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:42 dist-test-slave-8w9v krb5kdc[29809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:42 dist-test-slave-8w9v krb5kdc[29809](info): set up 2 sockets
Aug 12 14:26:42 dist-test-slave-8w9v krb5kdc[29809](info): commencing operation
krb5kdc: starting...
W20260812 14:26:43.275207 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:43 dist-test-slave-8w9v krb5kdc[29809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:26:43.402626 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:43.487238 29825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:43.487488 29825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:43.487530 29825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:43.488716 29825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:43.488788 29825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:43.488811 29825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:43.488832 29825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:43.488862 29825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:43.490312 29825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38727
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:43.490613 29825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:43.490844 29825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:43.495757 29834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:43.495821 29830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:43.495961 29825 server_base.cc:1034] running on GCE node
W20260812 14:26:43.495757 29831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:43.496325 29825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:43.496608 29825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:43.497756 29825 hybrid_clock.cc:648] HybridClock initialized: now 1786544803497734 us; error 37 us; skew 500 ppm
Aug 12 14:26:43 dist-test-slave-8w9v krb5kdc[29809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:43.500291 29825 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:26:43.500744 29825 webserver.cc:460] Webserver started at http://127.0.0.1:39271/ using document root <none> and password file <none>
I20260812 14:26:43.500984 29825 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:43.501031 29825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:43.501202 29825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:43.502280 29825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "79e769be6f024832922307816d21f605"
format_stamp: "Formatted at 2026-08-12 14:26:43 on dist-test-slave-8w9v"
server_key: "fb811c8b7027e09a66b75a22dd378086"
server_key_iv: "f3b765753b5cd2da062b4f7736dd32f8"
server_key_version: "encryptionkey@0"
I20260812 14:26:43.502656 29825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "79e769be6f024832922307816d21f605"
format_stamp: "Formatted at 2026-08-12 14:26:43 on dist-test-slave-8w9v"
server_key: "fb811c8b7027e09a66b75a22dd378086"
server_key_iv: "f3b765753b5cd2da062b4f7736dd32f8"
server_key_version: "encryptionkey@0"
I20260812 14:26:43.504029 29825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:26:43.504966 29842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:43.505158 29825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:43.505285 29825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "79e769be6f024832922307816d21f605"
format_stamp: "Formatted at 2026-08-12 14:26:43 on dist-test-slave-8w9v"
server_key: "fb811c8b7027e09a66b75a22dd378086"
server_key_iv: "f3b765753b5cd2da062b4f7736dd32f8"
server_key_version: "encryptionkey@0"
I20260812 14:26:43.505385 29825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:43.538385 29825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:43.540843 29825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:43.541043 29825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:43.546226 29825 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38727
I20260812 14:26:43.546289 29898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38727 every 8 connection(s)
I20260812 14:26:43.546723 29825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:43.547786 29899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:43.548952 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 29825
I20260812 14:26:43.549074 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:43.549278 5712 external_mini_cluster.cc:1428] Setting key d1ab36a15a0dcab04c9d7008f71daaac
I20260812 14:26:43.550770 29899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605: Bootstrap starting.
I20260812 14:26:43.551522 29899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:43.551849 29899 log.cc:824] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:43.552677 29899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605: No bootstrap required, opened a new log
I20260812 14:26:43.555173 29899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e769be6f024832922307816d21f605" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } }
I20260812 14:26:43.555361 29899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:43.555418 29899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e769be6f024832922307816d21f605, State: Initialized, Role: FOLLOWER
I20260812 14:26:43.555586 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:43.549825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39917 (local address 127.0.0.1:38727)
0812 14:26:43.550029 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:43.550037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:43.550064 (+ 27us) server_negotiation.cc:407] Connection header received
0812 14:26:43.550526 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:43.550533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:43.550581 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:43.550701 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:43.550995 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:43.552451 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:43.553296 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:43.553926 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:43.554040 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:43.554817 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:43.554823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:43.554824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:43.555038 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:43.555078 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:43.555093 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:43.555187 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:43.555268 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 14:26:43.555298 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260812 14:26:43.555557 29899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e769be6f024832922307816d21f605" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } }
I20260812 14:26:43.555656 29899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:43.555691 29899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:43.555738 29899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:43.556524 29899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e769be6f024832922307816d21f605" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } }
I20260812 14:26:43.556658 29899 leader_election.cc:302] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79e769be6f024832922307816d21f605; no voters:
I20260812 14:26:43.556779 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:26:43.556834 29899 leader_election.cc:288] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:43.556936 29905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:43.557120 29899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:43.557150 29905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [term 1 LEADER]: Becoming Leader. State: Replica: 79e769be6f024832922307816d21f605, State: Running, Role: LEADER
I20260812 14:26:43.557296 29905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e769be6f024832922307816d21f605" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } }
I20260812 14:26:43.557760 29907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79e769be6f024832922307816d21f605. Latest consensus state: current_term: 1 leader_uuid: "79e769be6f024832922307816d21f605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e769be6f024832922307816d21f605" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } } }
I20260812 14:26:43.557900 29907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:43.558131 29906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79e769be6f024832922307816d21f605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e769be6f024832922307816d21f605" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } } }
I20260812 14:26:43.558259 29906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:43.558384 29917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:43.558945 29917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:43.560599 29917 catalog_manager.cc:1347] Generated new cluster ID: 7e5710128da94d8fbfd4444ff47ad748
I20260812 14:26:43.560678 29917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:43.562268 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:43.558636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51947 (local address 127.0.0.1:38727)
0812 14:26:43.558671 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:43.558673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:43.558810 (+ 137us) server_negotiation.cc:407] Connection header received
0812 14:26:43.558913 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:43.558914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:43.558927 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:43.558982 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:43.559250 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:43.559985 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:43.561002 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:43.561324 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:43.561395 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:43.562047 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:43.562048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:43.562049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:43.562053 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:43.562069 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:43.562073 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:43.562129 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:43.562186 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 14:26:43.562196 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:26:43.576407 29917 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:43.576979 29917 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:43.583949 29917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79e769be6f024832922307816d21f605: Generated new TSK 0
I20260812 14:26:43.584210 29917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:43.587777 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 29825
W20260812 14:26:43.597023 29807 connection.cc:535] client connection to 127.0.0.1:38727 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38727): Connection reset by peer (error 104)
2026-08-12T14:26:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:43 dist-test-slave-8w9v krb5kdc[29938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:43 dist-test-slave-8w9v krb5kdc[29938](info): set up 2 sockets
Aug 12 14:26:43 dist-test-slave-8w9v krb5kdc[29938](info): commencing operation
krb5kdc: starting...
W20260812 14:26:44.769623 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:44 dist-test-slave-8w9v krb5kdc[29938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:26:44.899326 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:44.985283 29953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:44.985508 29953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:44.985534 29953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:44.986616 29953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:44.986658 29953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:44.986676 29953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:44.986690 29953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:44.986706 29953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:44.988219 29953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:44.988440 29953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:44.988631 29953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:44.993664 29959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:44.993762 29958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:44.993635 29963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:44.993882 29953 server_base.cc:1034] running on GCE node
I20260812 14:26:44.994131 29953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:44.994378 29953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:44.995523 29953 hybrid_clock.cc:648] HybridClock initialized: now 1786544804995508 us; error 28 us; skew 500 ppm
Aug 12 14:26:44 dist-test-slave-8w9v krb5kdc[29938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:45.002337 29953 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:26:45.002842 29953 webserver.cc:460] Webserver started at http://127.0.0.1:42533/ using document root <none> and password file <none>
I20260812 14:26:45.003093 29953 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:45.003165 29953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:45.003319 29953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:45.004510 29953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "30f3fdf8ff9848f08da63780d1fb944e"
format_stamp: "Formatted at 2026-08-12 14:26:44 on dist-test-slave-8w9v"
server_key: "4278ea70f6966e25411c73de2963a27f"
server_key_iv: "11bfa830c17d4e0cb664f98868628c97"
server_key_version: "encryptionkey@0"
I20260812 14:26:45.004884 29953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "30f3fdf8ff9848f08da63780d1fb944e"
format_stamp: "Formatted at 2026-08-12 14:26:44 on dist-test-slave-8w9v"
server_key: "4278ea70f6966e25411c73de2963a27f"
server_key_iv: "11bfa830c17d4e0cb664f98868628c97"
server_key_version: "encryptionkey@0"
I20260812 14:26:45.006258 29953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:26:45.007085 29972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:45.007289 29953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:45.007375 29953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "30f3fdf8ff9848f08da63780d1fb944e"
format_stamp: "Formatted at 2026-08-12 14:26:44 on dist-test-slave-8w9v"
server_key: "4278ea70f6966e25411c73de2963a27f"
server_key_iv: "11bfa830c17d4e0cb664f98868628c97"
server_key_version: "encryptionkey@0"
I20260812 14:26:45.007480 29953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:45.010987 29953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:45.013587 29953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:45.013756 29953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:45.018218 29953 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34057
I20260812 14:26:45.018253 30030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34057 every 8 connection(s)
I20260812 14:26:45.018620 29953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:45.019626 30031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:45.022049 30031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e: Bootstrap starting.
I20260812 14:26:45.022604 30031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:45.022852 30031 log.cc:824] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:45.023564 30031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e: No bootstrap required, opened a new log
I20260812 14:26:45.025233 30031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f3fdf8ff9848f08da63780d1fb944e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } }
I20260812 14:26:45.025453 30031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:45.025497 30031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f3fdf8ff9848f08da63780d1fb944e, State: Initialized, Role: FOLLOWER
I20260812 14:26:45.025498 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 29953
I20260812 14:26:45.025617 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:45.025669 30031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f3fdf8ff9848f08da63780d1fb944e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } }
I20260812 14:26:45.025761 30031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:45.025794 30031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:45.025831 5712 external_mini_cluster.cc:1428] Setting key 6852c05adcbc440f6b3659f403498855
I20260812 14:26:45.025846 30031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:45.026588 30031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f3fdf8ff9848f08da63780d1fb944e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } }
I20260812 14:26:45.026698 30031 leader_election.cc:302] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30f3fdf8ff9848f08da63780d1fb944e; no voters:
I20260812 14:26:45.026898 30031 leader_election.cc:288] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:45.026921 30037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:45.027110 30037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [term 1 LEADER]: Becoming Leader. State: Replica: 30f3fdf8ff9848f08da63780d1fb944e, State: Running, Role: LEADER
I20260812 14:26:45.027156 30031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:45.027240 30037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f3fdf8ff9848f08da63780d1fb944e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } }
I20260812 14:26:45.027971 30038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30f3fdf8ff9848f08da63780d1fb944e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f3fdf8ff9848f08da63780d1fb944e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } } }
I20260812 14:26:45.028168 30038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:45.028155 30039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30f3fdf8ff9848f08da63780d1fb944e. Latest consensus state: current_term: 1 leader_uuid: "30f3fdf8ff9848f08da63780d1fb944e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f3fdf8ff9848f08da63780d1fb944e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } } }
I20260812 14:26:45.028424 30039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:45.028605 30036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:45.026692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51401 (local address 127.0.0.1:34057)
0812 14:26:45.026910 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:45.026919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:45.026955 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:26:45.027081 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:45.027088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:45.027118 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:45.027264 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:45.027927 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:45.027931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:45.027932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:45.027936 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:45.027986 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:45.028005 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:45.028086 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:45.028302 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 14:26:45.028353 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":103,"threads_started":1}
I20260812 14:26:45.028652 30051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:45.029386 30051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:45.029688 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:26:45.031145 30051 catalog_manager.cc:1347] Generated new cluster ID: cca9b7e1dc5844658aaa49bd7a246394
I20260812 14:26:45.031195 30051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:45.033229 30036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:45.032205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37527 (local address 127.0.0.1:34057)
0812 14:26:45.032246 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:45.032248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:45.032258 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:26:45.032280 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:45.032281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:45.032287 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:45.032344 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:45.033008 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:45.033009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:45.033009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:45.033014 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:45.033031 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:45.033036 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:45.033101 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:45.033153 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 14:26:45.033166 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:26:45.037763 30051 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:45.037946 30051 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:45.047842 30051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30f3fdf8ff9848f08da63780d1fb944e: Generated new TSK 0
I20260812 14:26:45.048081 30051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:45.051128 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 29953
2026-08-12T14:26:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1464 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:45 dist-test-slave-8w9v krb5kdc[30066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:45 dist-test-slave-8w9v krb5kdc[30066](info): set up 2 sockets
Aug 12 14:26:45 dist-test-slave-8w9v krb5kdc[30066](info): commencing operation
krb5kdc: starting...
W20260812 14:26:46.256382 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:46 dist-test-slave-8w9v krb5kdc[30066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:26:46.387722 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:46.471056 30079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:46.471309 30079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:46.471347 30079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:46.472512 30079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:46.472589 30079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:46.472615 30079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:46.472635 30079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:46.472664 30079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:46.474169 30079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:46.474470 30079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:46.474706 30079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:26:46.479794 30079 server_base.cc:1034] running on GCE node
W20260812 14:26:46.479748 30085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:46.479745 30089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:46.479736 30087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:46.480315 30079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:46.480625 30079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:46.481796 30079 hybrid_clock.cc:648] HybridClock initialized: now 1786544806481792 us; error 38 us; skew 500 ppm
Aug 12 14:26:46 dist-test-slave-8w9v krb5kdc[30066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:46.484368 30079 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:26:46.484866 30079 webserver.cc:460] Webserver started at http://127.0.0.1:38451/ using document root <none> and password file <none>
I20260812 14:26:46.485105 30079 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:46.485177 30079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:46.485340 30079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:46.486495 30079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "207700b1809b4fd48821a851b77f25fb"
format_stamp: "Formatted at 2026-08-12 14:26:46 on dist-test-slave-8w9v"
server_key: "76dbc6c6a3362f59a3fce345370d92d2"
server_key_iv: "17bc12511f865f60253919e196f35f13"
server_key_version: "encryptionkey@0"
I20260812 14:26:46.486877 30079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "207700b1809b4fd48821a851b77f25fb"
format_stamp: "Formatted at 2026-08-12 14:26:46 on dist-test-slave-8w9v"
server_key: "76dbc6c6a3362f59a3fce345370d92d2"
server_key_iv: "17bc12511f865f60253919e196f35f13"
server_key_version: "encryptionkey@0"
I20260812 14:26:46.488281 30079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:26:46.489060 30100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:46.489253 30079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:46.489346 30079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "207700b1809b4fd48821a851b77f25fb"
format_stamp: "Formatted at 2026-08-12 14:26:46 on dist-test-slave-8w9v"
server_key: "76dbc6c6a3362f59a3fce345370d92d2"
server_key_iv: "17bc12511f865f60253919e196f35f13"
server_key_version: "encryptionkey@0"
I20260812 14:26:46.489403 30079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:46.505751 30079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:46.507719 30079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:46.507835 30079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:46.511925 30079 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35027
I20260812 14:26:46.512022 30162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35027 every 8 connection(s)
I20260812 14:26:46.512384 30079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:46.513455 30164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:46.513558 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 30079
I20260812 14:26:46.513693 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:46.513902 5712 external_mini_cluster.cc:1428] Setting key 5cf1ecec891c057389d6c96f1d27b8f8
I20260812 14:26:46.516959 30164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb: Bootstrap starting.
I20260812 14:26:46.517748 30164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:46.518075 30164 log.cc:824] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:46.518905 30164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb: No bootstrap required, opened a new log
Aug 12 14:26:46 dist-test-slave-8w9v krb5kdc[30066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 14:26:46.521692 30164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207700b1809b4fd48821a851b77f25fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } }
I20260812 14:26:46.521889 30164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:46.521931 30164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 207700b1809b4fd48821a851b77f25fb, State: Initialized, Role: FOLLOWER
I20260812 14:26:46.522140 30164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207700b1809b4fd48821a851b77f25fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } }
I20260812 14:26:46.522256 30164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:46.522312 30164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:46.522363 30164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:46.523011 30164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207700b1809b4fd48821a851b77f25fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } }
I20260812 14:26:46.523103 30164 leader_election.cc:302] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 207700b1809b4fd48821a851b77f25fb; no voters:
I20260812 14:26:46.523336 30164 leader_election.cc:288] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:46.523368 30169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:46.523486 30169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [term 1 LEADER]: Becoming Leader. State: Replica: 207700b1809b4fd48821a851b77f25fb, State: Running, Role: LEADER
I20260812 14:26:46.523622 30164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:46.523640 30169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207700b1809b4fd48821a851b77f25fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } }
I20260812 14:26:46.523989 30171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "207700b1809b4fd48821a851b77f25fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207700b1809b4fd48821a851b77f25fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } } }
I20260812 14:26:46.524094 30171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:46.523973 30169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 207700b1809b4fd48821a851b77f25fb. Latest consensus state: current_term: 1 leader_uuid: "207700b1809b4fd48821a851b77f25fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207700b1809b4fd48821a851b77f25fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } } }
I20260812 14:26:46.524428 30169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:46.524736 30176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:46.525341 30176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:46.525779 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:46.514469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34367 (local address 127.0.0.1:35027)
0812 14:26:46.514650 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:46.514658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:46.514685 (+ 27us) server_negotiation.cc:407] Connection header received
0812 14:26:46.515151 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:46.515159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:46.515206 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:46.515321 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:46.516225 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:46.517149 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:46.518235 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:46.518657 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:46.518747 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:46.521518 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:46.521522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:46.521522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:46.521534 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:46.523404 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:46.523839 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:46.523842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:46.523842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:46.523992 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:46.524280 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:46.524281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:46.524282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:46.524605 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:46.525238 (+ 633us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:46.525314 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 14:26:46.525471 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260812 14:26:46.526772 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:26:46.528223 30176 catalog_manager.cc:1347] Generated new cluster ID: e3b2e7d4f49d4958be94bddec0e8edf5
I20260812 14:26:46.528280 30176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:46.534508 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:46.528549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39817 (local address 127.0.0.1:35027)
0812 14:26:46.528584 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:46.528586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:46.528595 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:26:46.528615 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:46.528615 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:46.528623 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:46.528673 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:46.529194 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:46.529673 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:46.530362 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:46.530587 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:46.530675 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:46.531878 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:46.531879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:46.531880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:46.531886 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:46.533361 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:46.533716 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:46.533718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:46.533718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:46.533777 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:46.533981 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:46.533982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:46.533982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:46.534100 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:46.534202 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:46.534310 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 14:26:46.534388 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:26:46.544682 30176 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:46.545222 30176 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:46.554162 30176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 207700b1809b4fd48821a851b77f25fb: Generated new TSK 0
I20260812 14:26:46.554379 30176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:46.563527 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 30079
2026-08-12T14:26:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1510 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:46 dist-test-slave-8w9v krb5kdc[30199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:46 dist-test-slave-8w9v krb5kdc[30199](info): set up 2 sockets
Aug 12 14:26:46 dist-test-slave-8w9v krb5kdc[30199](info): commencing operation
krb5kdc: starting...
W20260812 14:26:47.689244 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:47 dist-test-slave-8w9v krb5kdc[30199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:26:47.827486 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:47.919679 30213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:47.920017 30213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:47.920063 30213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:47.922197 30213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:47.922287 30213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:47.922308 30213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:47.922389 30213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:47.922425 30213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:47.925014 30213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:47.925336 30213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:47.925664 30213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:47.931600 30219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:47.931600 30220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:47.931597 30222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:47.933916 30213 server_base.cc:1034] running on GCE node
I20260812 14:26:47.934185 30213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:47.934513 30213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:47.935717 30213 hybrid_clock.cc:648] HybridClock initialized: now 1786544807935685 us; error 48 us; skew 500 ppm
Aug 12 14:26:47 dist-test-slave-8w9v krb5kdc[30199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:47.939121 30213 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:26:47.939783 30213 webserver.cc:460] Webserver started at http://127.0.0.1:43455/ using document root <none> and password file <none>
I20260812 14:26:47.940038 30213 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:47.940109 30213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:47.940302 30213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:47.941417 30213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "7638da1cd6ad413c962473d291ac63f1"
format_stamp: "Formatted at 2026-08-12 14:26:47 on dist-test-slave-8w9v"
server_key: "56e7ee36c7e37fbf705c4eba8a7f4f4c"
server_key_iv: "30a9f69e8f7e1a1a28afd36bb2e7196c"
server_key_version: "encryptionkey@0"
I20260812 14:26:47.941799 30213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "7638da1cd6ad413c962473d291ac63f1"
format_stamp: "Formatted at 2026-08-12 14:26:47 on dist-test-slave-8w9v"
server_key: "56e7ee36c7e37fbf705c4eba8a7f4f4c"
server_key_iv: "30a9f69e8f7e1a1a28afd36bb2e7196c"
server_key_version: "encryptionkey@0"
I20260812 14:26:47.943470 30213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:26:47.944461 30232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:47.944671 30213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:47.944832 30213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "7638da1cd6ad413c962473d291ac63f1"
format_stamp: "Formatted at 2026-08-12 14:26:47 on dist-test-slave-8w9v"
server_key: "56e7ee36c7e37fbf705c4eba8a7f4f4c"
server_key_iv: "30a9f69e8f7e1a1a28afd36bb2e7196c"
server_key_version: "encryptionkey@0"
I20260812 14:26:47.944916 30213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:47.966666 30213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:47.969002 30213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:47.969170 30213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:47.973470 30213 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34197
I20260812 14:26:47.973521 30297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34197 every 8 connection(s)
I20260812 14:26:47.973922 30213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:47.974951 30298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:47.975339 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 30213
I20260812 14:26:47.975466 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:47.975666 5712 external_mini_cluster.cc:1428] Setting key 7ccdc41cedc955955a766490a0556566
I20260812 14:26:47.978308 30298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1: Bootstrap starting.
I20260812 14:26:47.979028 30298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:47.979370 30298 log.cc:824] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:47.980238 30298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1: No bootstrap required, opened a new log
I20260812 14:26:47.982025 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:47.976318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41483 (local address 127.0.0.1:34197)
0812 14:26:47.976546 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:47.976556 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:47.976592 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:26:47.977160 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:47.977169 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:47.977231 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:47.977394 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:47.977673 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:47.979063 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:47.980184 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:47.980638 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:47.980721 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:47.981447 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:47.981451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:47.981451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:47.981611 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:47.981643 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:47.981654 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:47.981706 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:47.981819 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 14:26:47.981872 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":97,"threads_started":1}
I20260812 14:26:47.982647 30298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7638da1cd6ad413c962473d291ac63f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34197 } }
I20260812 14:26:47.982846 30298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:47.982901 30298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7638da1cd6ad413c962473d291ac63f1, State: Initialized, Role: FOLLOWER
I20260812 14:26:47.983019 30298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7638da1cd6ad413c962473d291ac63f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34197 } }
I20260812 14:26:47.983065 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:26:47.983107 30298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:47.983153 30298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:47.983202 30298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:47.983763 30298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7638da1cd6ad413c962473d291ac63f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34197 } }
I20260812 14:26:47.983884 30298 leader_election.cc:302] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7638da1cd6ad413c962473d291ac63f1; no voters:
I20260812 14:26:47.984076 30298 leader_election.cc:288] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:47.984283 30310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:47.984385 30298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:47.984448 30310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [term 1 LEADER]: Becoming Leader. State: Replica: 7638da1cd6ad413c962473d291ac63f1, State: Running, Role: LEADER
I20260812 14:26:47.984550 30310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7638da1cd6ad413c962473d291ac63f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34197 } }
I20260812 14:26:47.985004 30310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7638da1cd6ad413c962473d291ac63f1. Latest consensus state: current_term: 1 leader_uuid: "7638da1cd6ad413c962473d291ac63f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7638da1cd6ad413c962473d291ac63f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34197 } } }
I20260812 14:26:47.985097 30310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:47.985018 30312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7638da1cd6ad413c962473d291ac63f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7638da1cd6ad413c962473d291ac63f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34197 } } }
I20260812 14:26:47.985149 30312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1 [sys.catalog]: This master's current role is: LEADER
W20260812 14:26:47.986411 30328 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:26:47.986471 30328 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:26:47.986519 30321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:47.986681 30321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:47.988260 30321 catalog_manager.cc:1347] Generated new cluster ID: 251edf08574a451d95746ae6b7b02660
I20260812 14:26:47.988260 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:47.984721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37321 (local address 127.0.0.1:34197)
0812 14:26:47.984772 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:47.984774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:47.984856 (+ 82us) server_negotiation.cc:407] Connection header received
0812 14:26:47.984958 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:47.984959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:47.984968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:47.984993 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:47.985202 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:47.985872 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:47.986901 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:47.987106 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:47.987217 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:47.987958 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:47.987960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:47.987960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:47.987965 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:47.987981 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:47.987986 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:47.988052 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:47.988152 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 14:26:47.988190 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:26:47.988339 30321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:47.993264 30321 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:47.993800 30321 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:47.996742 30321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7638da1cd6ad413c962473d291ac63f1: Generated new TSK 0
I20260812 14:26:47.996920 30321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:48.002637 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 30213
2026-08-12T14:26:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:48 dist-test-slave-8w9v krb5kdc[30334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:48 dist-test-slave-8w9v krb5kdc[30334](info): set up 2 sockets
Aug 12 14:26:48 dist-test-slave-8w9v krb5kdc[30334](info): commencing operation
krb5kdc: starting...
W20260812 14:26:49.133503 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:49 dist-test-slave-8w9v krb5kdc[30334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:26:49.263266 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42865
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:49.347755 30352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:49.347985 30352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:49.348008 30352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:49.349211 30352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:49.349257 30352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:49.349272 30352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:49.349284 30352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:49.349342 30352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:49.350835 30352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:49.351118 30352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:49.351367 30352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:49.356362 30360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:49.356393 30357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:49.356379 30358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:49.356456 30352 server_base.cc:1034] running on GCE node
I20260812 14:26:49.356765 30352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:49.357074 30352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:49.358222 30352 hybrid_clock.cc:648] HybridClock initialized: now 1786544809358210 us; error 28 us; skew 500 ppm
Aug 12 14:26:49 dist-test-slave-8w9v krb5kdc[30334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:49.360754 30352 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:26:49.361241 30352 webserver.cc:460] Webserver started at http://127.0.0.1:33541/ using document root <none> and password file <none>
I20260812 14:26:49.361511 30352 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:49.361585 30352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:49.361735 30352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:49.362840 30352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "a2931a15cb2c4904a899fef5789aefe6"
format_stamp: "Formatted at 2026-08-12 14:26:49 on dist-test-slave-8w9v"
server_key: "2426d2943271da5caedd40568add3057"
server_key_iv: "a62d7419494e0b679805ef5671839f28"
server_key_version: "encryptionkey@0"
I20260812 14:26:49.363427 30352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "a2931a15cb2c4904a899fef5789aefe6"
format_stamp: "Formatted at 2026-08-12 14:26:49 on dist-test-slave-8w9v"
server_key: "2426d2943271da5caedd40568add3057"
server_key_iv: "a62d7419494e0b679805ef5671839f28"
server_key_version: "encryptionkey@0"
I20260812 14:26:49.364948 30352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:26:49.365828 30369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:49.366021 30352 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:49.366138 30352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "a2931a15cb2c4904a899fef5789aefe6"
format_stamp: "Formatted at 2026-08-12 14:26:49 on dist-test-slave-8w9v"
server_key: "2426d2943271da5caedd40568add3057"
server_key_iv: "a62d7419494e0b679805ef5671839f28"
server_key_version: "encryptionkey@0"
I20260812 14:26:49.366217 30352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:49.369804 30352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:49.371712 30352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:49.371879 30352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:49.376286 30352 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42865
I20260812 14:26:49.376320 30437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42865 every 8 connection(s)
I20260812 14:26:49.376708 30352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:49.377785 30438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:49.378664 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 30352
I20260812 14:26:49.378813 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:49.379004 5712 external_mini_cluster.cc:1428] Setting key 0e0cf8be185bf07684f76a7ca0f71a7d
I20260812 14:26:49.381282 30438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6: Bootstrap starting.
I20260812 14:26:49.381619 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:49.379627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41205 (local address 127.0.0.1:42865)
0812 14:26:49.379805 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:49.379822 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:49.379847 (+ 25us) server_negotiation.cc:407] Connection header received
0812 14:26:49.380332 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:49.380340 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:49.380367 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:49.380494 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:49.381212 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:49.381215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:49.381216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:49.381219 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:49.381258 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:49.381269 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:49.381325 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:49.381420 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 14:26:49.381466 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260812 14:26:49.382073 30438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:49.382400 30438 log.cc:824] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:49.382736 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:26:49.383108 30438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6: No bootstrap required, opened a new log
I20260812 14:26:49.385345 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:49.384277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48467 (local address 127.0.0.1:42865)
0812 14:26:49.384322 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:49.384324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:49.384366 (+ 42us) server_negotiation.cc:407] Connection header received
0812 14:26:49.384458 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:49.384459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:49.384464 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:49.384518 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:49.385134 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:49.385135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:49.385136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:49.385142 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:49.385162 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:49.385167 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:49.385239 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:49.385274 (+ 35us) server_negotiation.cc:299] Negotiation successful
0812 14:26:49.385286 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:26:49.385502 30438 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2931a15cb2c4904a899fef5789aefe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42865 } }
I20260812 14:26:49.385672 30438 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:49.385717 30438 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2931a15cb2c4904a899fef5789aefe6, State: Initialized, Role: FOLLOWER
I20260812 14:26:49.385854 30438 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2931a15cb2c4904a899fef5789aefe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42865 } }
I20260812 14:26:49.385944 30438 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:49.385967 30438 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:49.385995 30438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:49.386514 30438 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2931a15cb2c4904a899fef5789aefe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42865 } }
I20260812 14:26:49.386600 30438 leader_election.cc:302] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2931a15cb2c4904a899fef5789aefe6; no voters:
I20260812 14:26:49.386731 30438 leader_election.cc:288] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:49.386791 30450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:49.386976 30450 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [term 1 LEADER]: Becoming Leader. State: Replica: a2931a15cb2c4904a899fef5789aefe6, State: Running, Role: LEADER
I20260812 14:26:49.387012 30438 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:49.387064 30450 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2931a15cb2c4904a899fef5789aefe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42865 } }
I20260812 14:26:49.387466 30452 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2931a15cb2c4904a899fef5789aefe6. Latest consensus state: current_term: 1 leader_uuid: "a2931a15cb2c4904a899fef5789aefe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2931a15cb2c4904a899fef5789aefe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42865 } } }
I20260812 14:26:49.387462 30451 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2931a15cb2c4904a899fef5789aefe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2931a15cb2c4904a899fef5789aefe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42865 } } }
I20260812 14:26:49.387578 30452 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:49.387586 30451 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:49.387809 30459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:49.388360 30459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:49.390005 30459 catalog_manager.cc:1347] Generated new cluster ID: d5c33bf6a6a34432b1bcb8bb3817a691
I20260812 14:26:49.390057 30459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:49.406200 30459 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:49.406402 30459 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:49.413048 30459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2931a15cb2c4904a899fef5789aefe6: Generated new TSK 0
I20260812 14:26:49.413265 30459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:49.415654 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 30352
2026-08-12T14:26:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:49 dist-test-slave-8w9v krb5kdc[30475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:49 dist-test-slave-8w9v krb5kdc[30475](info): set up 2 sockets
Aug 12 14:26:49 dist-test-slave-8w9v krb5kdc[30475](info): commencing operation
krb5kdc: starting...
W20260812 14:26:50.544407 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:50 dist-test-slave-8w9v krb5kdc[30475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:26:50.675220 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42995
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:50.760344 30493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:50.760560 30493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:50.760583 30493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:50.761823 30493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:50.761873 30493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:50.761893 30493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:50.761906 30493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:50.761919 30493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:50.763356 30493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:50.763556 30493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:50.763738 30493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:50.768832 30503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:50.768872 30501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:50.768918 30493 server_base.cc:1034] running on GCE node
W20260812 14:26:50.768823 30499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:50.769321 30493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:50.769649 30493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:50.770794 30493 hybrid_clock.cc:648] HybridClock initialized: now 1786544810770786 us; error 31 us; skew 500 ppm
Aug 12 14:26:50 dist-test-slave-8w9v krb5kdc[30475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:50.776791 30493 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:26:50.777293 30493 webserver.cc:460] Webserver started at http://127.0.0.1:40589/ using document root <none> and password file <none>
I20260812 14:26:50.777524 30493 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:50.777567 30493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:50.777661 30493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:50.778716 30493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "6488dd4cccd3455e9ba7ce2d2e41b1b4"
format_stamp: "Formatted at 2026-08-12 14:26:50 on dist-test-slave-8w9v"
server_key: "2908cfffb7e219ad1352225c7a2ecbda"
server_key_iv: "bbbd30662f7a90a601e2b0e8edca76c0"
server_key_version: "encryptionkey@0"
I20260812 14:26:50.779091 30493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "6488dd4cccd3455e9ba7ce2d2e41b1b4"
format_stamp: "Formatted at 2026-08-12 14:26:50 on dist-test-slave-8w9v"
server_key: "2908cfffb7e219ad1352225c7a2ecbda"
server_key_iv: "bbbd30662f7a90a601e2b0e8edca76c0"
server_key_version: "encryptionkey@0"
I20260812 14:26:50.780467 30493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:26:50.781226 30510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:50.781473 30493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:50.781555 30493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "6488dd4cccd3455e9ba7ce2d2e41b1b4"
format_stamp: "Formatted at 2026-08-12 14:26:50 on dist-test-slave-8w9v"
server_key: "2908cfffb7e219ad1352225c7a2ecbda"
server_key_iv: "bbbd30662f7a90a601e2b0e8edca76c0"
server_key_version: "encryptionkey@0"
I20260812 14:26:50.781649 30493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:50.803781 30493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:50.805855 30493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:50.806018 30493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:50.810523 30493 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42995
I20260812 14:26:50.810586 30568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42995 every 8 connection(s)
I20260812 14:26:50.810961 30493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:50.811992 30569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:50.814412 30569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4: Bootstrap starting.
I20260812 14:26:50.815053 30569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:50.815362 30569 log.cc:824] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:50.816190 30569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4: No bootstrap required, opened a new log
I20260812 14:26:50.817804 30569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6488dd4cccd3455e9ba7ce2d2e41b1b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } }
I20260812 14:26:50.817982 30569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:50.818053 30569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6488dd4cccd3455e9ba7ce2d2e41b1b4, State: Initialized, Role: FOLLOWER
I20260812 14:26:50.818233 30569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6488dd4cccd3455e9ba7ce2d2e41b1b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } }
I20260812 14:26:50.818320 30569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:50.818365 30569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:50.818414 30569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:50.818976 30569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6488dd4cccd3455e9ba7ce2d2e41b1b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } }
I20260812 14:26:50.819100 30569 leader_election.cc:302] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6488dd4cccd3455e9ba7ce2d2e41b1b4; no voters:
I20260812 14:26:50.819300 30569 leader_election.cc:288] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:50.819348 30573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:50.819557 30573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [term 1 LEADER]: Becoming Leader. State: Replica: 6488dd4cccd3455e9ba7ce2d2e41b1b4, State: Running, Role: LEADER
I20260812 14:26:50.819584 30569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:50.819649 30573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6488dd4cccd3455e9ba7ce2d2e41b1b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } }
I20260812 14:26:50.820156 30574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6488dd4cccd3455e9ba7ce2d2e41b1b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6488dd4cccd3455e9ba7ce2d2e41b1b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } } }
I20260812 14:26:50.820173 30575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6488dd4cccd3455e9ba7ce2d2e41b1b4. Latest consensus state: current_term: 1 leader_uuid: "6488dd4cccd3455e9ba7ce2d2e41b1b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6488dd4cccd3455e9ba7ce2d2e41b1b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } } }
I20260812 14:26:50.820292 30574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:50.820303 30575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:50.821048 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 30493
I20260812 14:26:50.821197 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:50.821465 5712 external_mini_cluster.cc:1428] Setting key 0322e5d59dc83387397808765004e1f0
W20260812 14:26:50.821995 30594 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:26:50.822075 30594 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:26:50.822129 30581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:50.822297 30581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:50.823947 30581 catalog_manager.cc:1347] Generated new cluster ID: 105ee7df148c41489a002f0f726dec9c
I20260812 14:26:50.823997 30581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:26:50 dist-test-slave-8w9v krb5kdc[30475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 14:26:50.833735 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:50.822192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:53201 (local address 127.0.0.1:42995)
0812 14:26:50.822358 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:50.822367 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:50.822401 (+ 34us) server_negotiation.cc:407] Connection header received
0812 14:26:50.822535 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:50.822543 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:50.822602 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:50.822750 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:50.823503 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:50.824961 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:50.826006 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:50.826660 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:50.826777 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:50.829772 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:50.829776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:50.829777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:50.829790 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:50.831865 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:50.832374 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:50.832377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:50.832378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:50.832500 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:50.832757 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:50.832758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:50.832758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:50.833067 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:50.833238 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:50.833393 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 14:26:50.833496 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260812 14:26:50.834916 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:26:50.844235 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:50.836566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:56693 (local address 127.0.0.1:42995)
0812 14:26:50.836622 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:50.836624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:50.836842 (+ 218us) server_negotiation.cc:407] Connection header received
0812 14:26:50.836873 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:50.836874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:50.836885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:50.836954 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:50.837566 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:50.838395 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:50.839383 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:50.839724 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:50.839801 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:50.841223 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:50.841225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:50.841225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:50.841233 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:50.842929 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:50.843344 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:50.843345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:50.843346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:50.843409 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:50.843645 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:50.843646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:50.843646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:50.843787 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:50.843894 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:50.844040 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 14:26:50.844108 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:26:50.851178 30581 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:50.851776 30581 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:50.855597 30581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6488dd4cccd3455e9ba7ce2d2e41b1b4: Generated new TSK 0
I20260812 14:26:50.855795 30581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:50.861676 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 30493
W20260812 14:26:50.870643 30473 connection.cc:535] client connection to 127.0.0.1:42995 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42995): Connection reset by peer (error 104)
2026-08-12T14:26:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:50 dist-test-slave-8w9v krb5kdc[30608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:50 dist-test-slave-8w9v krb5kdc[30608](info): set up 2 sockets
Aug 12 14:26:50 dist-test-slave-8w9v krb5kdc[30608](info): commencing operation
krb5kdc: starting...
W20260812 14:26:51.979338 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:52 dist-test-slave-8w9v krb5kdc[30608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:26:52.105312 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:52.189021 30621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:52.189232 30621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:52.189255 30621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:52.190327 30621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:52.190364 30621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:52.190379 30621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:52.190398 30621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:52.190410 30621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:52.191814 30621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:52.192009 30621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:52.192198 30621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:52.197172 30631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:52.197199 30628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:52.197340 30621 server_base.cc:1034] running on GCE node
W20260812 14:26:52.197499 30627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:52.197768 30621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:52.198079 30621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:52.199254 30621 hybrid_clock.cc:648] HybridClock initialized: now 1786544812199234 us; error 39 us; skew 500 ppm
Aug 12 14:26:52 dist-test-slave-8w9v krb5kdc[30608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:52.201882 30621 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:26:52.202327 30621 webserver.cc:460] Webserver started at http://127.0.0.1:38623/ using document root <none> and password file <none>
I20260812 14:26:52.202584 30621 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:52.202656 30621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:52.202804 30621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:52.203902 30621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "6f43e4b43ce04ddd96a8353e1821862c"
format_stamp: "Formatted at 2026-08-12 14:26:52 on dist-test-slave-8w9v"
server_key: "1704bf011c1484fa7eb4aac7f8672920"
server_key_iv: "5caed74166b021e43d426a0b62c57818"
server_key_version: "encryptionkey@0"
I20260812 14:26:52.204304 30621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "6f43e4b43ce04ddd96a8353e1821862c"
format_stamp: "Formatted at 2026-08-12 14:26:52 on dist-test-slave-8w9v"
server_key: "1704bf011c1484fa7eb4aac7f8672920"
server_key_iv: "5caed74166b021e43d426a0b62c57818"
server_key_version: "encryptionkey@0"
I20260812 14:26:52.205664 30621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:26:52.206379 30638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:52.206559 30621 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:52.206670 30621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "6f43e4b43ce04ddd96a8353e1821862c"
format_stamp: "Formatted at 2026-08-12 14:26:52 on dist-test-slave-8w9v"
server_key: "1704bf011c1484fa7eb4aac7f8672920"
server_key_iv: "5caed74166b021e43d426a0b62c57818"
server_key_version: "encryptionkey@0"
I20260812 14:26:52.206748 30621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:52.213940 30621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:52.215693 30621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:52.215850 30621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:52.219825 30621 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37139
I20260812 14:26:52.219872 30698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37139 every 8 connection(s)
I20260812 14:26:52.220292 30621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:52.220754 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 30621
I20260812 14:26:52.220901 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:52.221131 5712 external_mini_cluster.cc:1428] Setting key 3d2e952b363eaed0549e80edd24d030a
I20260812 14:26:52.221496 30700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:52.223824 30700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c: Bootstrap starting.
I20260812 14:26:52.224525 30700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:52.224859 30700 log.cc:824] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:52.225582 30700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c: No bootstrap required, opened a new log
I20260812 14:26:52.227383 30700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43e4b43ce04ddd96a8353e1821862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37139 } }
I20260812 14:26:52.227551 30700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:52.227600 30700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f43e4b43ce04ddd96a8353e1821862c, State: Initialized, Role: FOLLOWER
I20260812 14:26:52.227757 30700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43e4b43ce04ddd96a8353e1821862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37139 } }
I20260812 14:26:52.227859 30700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:52.227918 30700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:52.227974 30700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:52.228597 30700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43e4b43ce04ddd96a8353e1821862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37139 } }
I20260812 14:26:52.228721 30700 leader_election.cc:302] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f43e4b43ce04ddd96a8353e1821862c; no voters:
I20260812 14:26:52.228916 30700 leader_election.cc:288] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:52.228955 30705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:52.229157 30705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [term 1 LEADER]: Becoming Leader. State: Replica: 6f43e4b43ce04ddd96a8353e1821862c, State: Running, Role: LEADER
I20260812 14:26:52.229197 30700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:52.229439 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:52.221648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:47555 (local address 127.0.0.1:37139)
0812 14:26:52.221844 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:52.221852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:52.224358 (+ 2506us) server_negotiation.cc:407] Connection header received
0812 14:26:52.225042 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:52.225051 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:52.225118 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:52.225278 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:52.225531 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:52.226470 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:52.227267 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:52.227765 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:52.227883 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:52.228673 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:52.228678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:52.228678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:52.228918 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:52.228961 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:52.228977 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:52.229071 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:52.229183 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 14:26:52.229213 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260812 14:26:52.229277 30705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43e4b43ce04ddd96a8353e1821862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37139 } }
I20260812 14:26:52.229966 30708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f43e4b43ce04ddd96a8353e1821862c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43e4b43ce04ddd96a8353e1821862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37139 } } }
I20260812 14:26:52.230064 30708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:52.230252 30715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:52.230732 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:26:52.230751 30715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:52.232209 30715 catalog_manager.cc:1347] Generated new cluster ID: ab0baa82dd244bd289387c18d6073191
I20260812 14:26:52.232262 30715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:52.232494 30706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f43e4b43ce04ddd96a8353e1821862c. Latest consensus state: current_term: 1 leader_uuid: "6f43e4b43ce04ddd96a8353e1821862c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43e4b43ce04ddd96a8353e1821862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37139 } } }
I20260812 14:26:52.232566 30706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:52.236137 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:52.232415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:46717 (local address 127.0.0.1:37139)
0812 14:26:52.232451 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:52.232452 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:52.232759 (+ 307us) server_negotiation.cc:407] Connection header received
0812 14:26:52.232813 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:52.232814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:52.232824 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:52.232880 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:52.233053 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:52.233756 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:52.234721 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:52.235032 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:52.235066 (+ 34us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:52.235859 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:52.235861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:52.235861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:52.235865 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:52.235877 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:52.235881 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:52.235910 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:52.236035 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 14:26:52.236046 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:26:52.239171 30715 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:52.239861 30715 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:52.254824 30715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f43e4b43ce04ddd96a8353e1821862c: Generated new TSK 0
I20260812 14:26:52.255076 30715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:52.262132 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 30621
W20260812 14:26:52.271325 30605 connection.cc:535] client connection to 127.0.0.1:37139 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37139): Connection reset by peer (error 104)
2026-08-12T14:26:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:52 dist-test-slave-8w9v krb5kdc[30742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:52 dist-test-slave-8w9v krb5kdc[30742](info): set up 2 sockets
Aug 12 14:26:52 dist-test-slave-8w9v krb5kdc[30742](info): commencing operation
krb5kdc: starting...
W20260812 14:26:53.387988 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:53 dist-test-slave-8w9v krb5kdc[30742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:26:53.515190 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42099
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:53.601193 30757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:53.601435 30757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:53.601476 30757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:53.602608 30757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:53.602672 30757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:53.602695 30757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:53.602715 30757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:53.602744 30757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:53.604256 30757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:53.604540 30757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:53.604772 30757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:26:53.609781 30757 server_base.cc:1034] running on GCE node
W20260812 14:26:53.609825 30763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:53.609822 30762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:53.610009 30765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:53.610299 30757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:53.610617 30757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:53.611793 30757 hybrid_clock.cc:648] HybridClock initialized: now 1786544813611786 us; error 35 us; skew 500 ppm
Aug 12 14:26:53 dist-test-slave-8w9v krb5kdc[30742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:53.614428 30757 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:26:53.614907 30757 webserver.cc:460] Webserver started at http://127.0.0.1:36367/ using document root <none> and password file <none>
I20260812 14:26:53.615160 30757 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:53.615231 30757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:53.615379 30757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:53.616564 30757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "ad326cf622784e01a500692503167a42"
format_stamp: "Formatted at 2026-08-12 14:26:53 on dist-test-slave-8w9v"
server_key: "bd4c26860e0d56308b82999a820d098a"
server_key_iv: "cd2a2fade76b64c0ffefba00465c1aa9"
server_key_version: "encryptionkey@0"
I20260812 14:26:53.616933 30757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "ad326cf622784e01a500692503167a42"
format_stamp: "Formatted at 2026-08-12 14:26:53 on dist-test-slave-8w9v"
server_key: "bd4c26860e0d56308b82999a820d098a"
server_key_iv: "cd2a2fade76b64c0ffefba00465c1aa9"
server_key_version: "encryptionkey@0"
I20260812 14:26:53.618278 30757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 14:26:53.618993 30774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:53.619164 30757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:53.619259 30757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "ad326cf622784e01a500692503167a42"
format_stamp: "Formatted at 2026-08-12 14:26:53 on dist-test-slave-8w9v"
server_key: "bd4c26860e0d56308b82999a820d098a"
server_key_iv: "cd2a2fade76b64c0ffefba00465c1aa9"
server_key_version: "encryptionkey@0"
I20260812 14:26:53.619345 30757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:53.622700 30757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:53.625113 30757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:53.625275 30757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:53.629537 30757 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42099
I20260812 14:26:53.629566 30836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42099 every 8 connection(s)
I20260812 14:26:53.629904 30757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:53.630486 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 30757
I20260812 14:26:53.630635 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:53.630893 5712 external_mini_cluster.cc:1428] Setting key 97660cac24277c1aa1a8b3b0a82723a0
I20260812 14:26:53.631237 30837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:53.633716 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:53.631437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:35609 (local address 127.0.0.1:42099)
0812 14:26:53.631654 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:53.631662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:53.631692 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:26:53.632400 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:53.632408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:53.632440 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:53.632596 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:53.633301 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:53.633305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:53.633305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:53.633309 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:53.633344 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:53.633354 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:53.633433 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:53.633517 (+ 84us) server_negotiation.cc:299] Negotiation successful
0812 14:26:53.633563 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
I20260812 14:26:53.634817 30837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42: Bootstrap starting.
I20260812 14:26:53.634833 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:26:53.635462 30837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:53.635814 30837 log.cc:824] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:53.636721 30837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42: No bootstrap required, opened a new log
I20260812 14:26:53.637574 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:53.636329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:50279 (local address 127.0.0.1:42099)
0812 14:26:53.636369 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:53.636371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:53.636520 (+ 149us) server_negotiation.cc:407] Connection header received
0812 14:26:53.636551 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:53.636552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:53.636557 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:53.636621 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:53.637219 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:53.637220 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:53.637221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:53.637226 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:53.637242 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:53.637247 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:53.637318 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:53.637357 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 14:26:53.637370 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:26:53.639101 30837 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad326cf622784e01a500692503167a42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42099 } }
I20260812 14:26:53.639240 30837 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:53.639263 30837 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad326cf622784e01a500692503167a42, State: Initialized, Role: FOLLOWER
I20260812 14:26:53.639415 30837 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad326cf622784e01a500692503167a42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42099 } }
I20260812 14:26:53.639520 30837 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:53.639557 30837 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:53.639611 30837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:53.640241 30837 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad326cf622784e01a500692503167a42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42099 } }
I20260812 14:26:53.640332 30837 leader_election.cc:302] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad326cf622784e01a500692503167a42; no voters:
I20260812 14:26:53.640473 30837 leader_election.cc:288] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:53.640528 30850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:53.640697 30850 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [term 1 LEADER]: Becoming Leader. State: Replica: ad326cf622784e01a500692503167a42, State: Running, Role: LEADER
I20260812 14:26:53.640759 30837 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:53.640830 30850 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad326cf622784e01a500692503167a42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42099 } }
I20260812 14:26:53.641273 30851 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad326cf622784e01a500692503167a42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad326cf622784e01a500692503167a42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42099 } } }
I20260812 14:26:53.641281 30852 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad326cf622784e01a500692503167a42. Latest consensus state: current_term: 1 leader_uuid: "ad326cf622784e01a500692503167a42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad326cf622784e01a500692503167a42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42099 } } }
I20260812 14:26:53.641466 30851 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:53.641474 30852 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:53.641723 30860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:53.642248 30860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:53.643810 30860 catalog_manager.cc:1347] Generated new cluster ID: b464305c9f414965b039eed3e04a8df4
I20260812 14:26:53.643860 30860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:53.656509 30860 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:53.656689 30860 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:53.660466 30860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad326cf622784e01a500692503167a42: Generated new TSK 0
I20260812 14:26:53.660674 30860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:53.663089 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 30757
2026-08-12T14:26:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:53 dist-test-slave-8w9v krb5kdc[30876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:53 dist-test-slave-8w9v krb5kdc[30876](info): set up 2 sockets
Aug 12 14:26:53 dist-test-slave-8w9v krb5kdc[30876](info): commencing operation
krb5kdc: starting...
W20260812 14:26:54.797993 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:54 dist-test-slave-8w9v krb5kdc[30876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:26:54.926887 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:55.012341 30894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:55.012619 30894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:55.012665 30894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:55.013899 30894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:55.013983 30894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:55.014012 30894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:55.014045 30894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:55.014070 30894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:55.015663 30894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45193
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:55.015974 30894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:55.016310 30894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:55.021574 30899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:55.021574 30900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:55.021625 30903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:55.021791 30894 server_base.cc:1034] running on GCE node
I20260812 14:26:55.022145 30894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:55.022521 30894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:55.023691 30894 hybrid_clock.cc:648] HybridClock initialized: now 1786544815023675 us; error 28 us; skew 500 ppm
Aug 12 14:26:55 dist-test-slave-8w9v krb5kdc[30876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:55.026335 30894 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:26:55.026808 30894 webserver.cc:460] Webserver started at http://127.0.0.1:37941/ using document root <none> and password file <none>
I20260812 14:26:55.027074 30894 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:55.027146 30894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:55.027302 30894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:55.028491 30894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "9e90d7c924434bf694b098ea773f2b7e"
format_stamp: "Formatted at 2026-08-12 14:26:55 on dist-test-slave-8w9v"
server_key: "34a041f74e211a8c3aae763d413354a3"
server_key_iv: "54098c9bd533ca4c9d44834bf027e3b5"
server_key_version: "encryptionkey@0"
I20260812 14:26:55.028865 30894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "9e90d7c924434bf694b098ea773f2b7e"
format_stamp: "Formatted at 2026-08-12 14:26:55 on dist-test-slave-8w9v"
server_key: "34a041f74e211a8c3aae763d413354a3"
server_key_iv: "54098c9bd533ca4c9d44834bf027e3b5"
server_key_version: "encryptionkey@0"
I20260812 14:26:55.030242 30894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:55.031028 30911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:55.031211 30894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:26:55.031329 30894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "9e90d7c924434bf694b098ea773f2b7e"
format_stamp: "Formatted at 2026-08-12 14:26:55 on dist-test-slave-8w9v"
server_key: "34a041f74e211a8c3aae763d413354a3"
server_key_iv: "54098c9bd533ca4c9d44834bf027e3b5"
server_key_version: "encryptionkey@0"
I20260812 14:26:55.031410 30894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:55.040352 30894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:55.042215 30894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:55.042383 30894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:55.046779 30894 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45193
I20260812 14:26:55.046813 30971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45193 every 8 connection(s)
I20260812 14:26:55.047211 30894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:55.048278 30972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:55.050621 30972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e: Bootstrap starting.
I20260812 14:26:55.051179 30972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:55.051446 30972 log.cc:824] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:55.052225 30972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e: No bootstrap required, opened a new log
I20260812 14:26:55.052850 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 30894
I20260812 14:26:55.053013 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:55.053218 5712 external_mini_cluster.cc:1428] Setting key 1e8a6bdd640b30a610845c176b197e89
I20260812 14:26:55.054517 30972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e90d7c924434bf694b098ea773f2b7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45193 } }
I20260812 14:26:55.054700 30972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:55.054739 30972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e90d7c924434bf694b098ea773f2b7e, State: Initialized, Role: FOLLOWER
I20260812 14:26:55.054904 30972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e90d7c924434bf694b098ea773f2b7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45193 } }
I20260812 14:26:55.054975 30972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:55.054996 30972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:55.055059 30972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:55.056224 30972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e90d7c924434bf694b098ea773f2b7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45193 } }
I20260812 14:26:55.056336 30972 leader_election.cc:302] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e90d7c924434bf694b098ea773f2b7e; no voters:
I20260812 14:26:55.056515 30972 leader_election.cc:288] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:55.056573 30979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:55.056754 30979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [term 1 LEADER]: Becoming Leader. State: Replica: 9e90d7c924434bf694b098ea773f2b7e, State: Running, Role: LEADER
I20260812 14:26:55.056753 30972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:55.056936 30979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e90d7c924434bf694b098ea773f2b7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45193 } }
I20260812 14:26:55.058615 30980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e90d7c924434bf694b098ea773f2b7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e90d7c924434bf694b098ea773f2b7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45193 } } }
I20260812 14:26:55.058749 30980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [sys.catalog]: This master's current role is: LEADER
W20260812 14:26:55.058872 30995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:26:55.058976 30995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:26:55.058947 30981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e90d7c924434bf694b098ea773f2b7e. Latest consensus state: current_term: 1 leader_uuid: "9e90d7c924434bf694b098ea773f2b7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e90d7c924434bf694b098ea773f2b7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45193 } } }
I20260812 14:26:55.059088 30981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:55.060303 30997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:55.060530 30997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 14:26:55 dist-test-slave-8w9v krb5kdc[30876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 14:26:55.062294 30997 catalog_manager.cc:1347] Generated new cluster ID: 85420d375531470883cbee8d2fdffc07
I20260812 14:26:55.062350 30997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:55.066771 30977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:55.053815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:59681 (local address 127.0.0.1:45193)
0812 14:26:55.054043 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:55.054054 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:55.054093 (+ 39us) server_negotiation.cc:407] Connection header received
0812 14:26:55.054471 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:55.054480 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:55.054549 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:55.054707 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:55.055538 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:55.057041 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:55.058159 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:55.058736 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:55.058853 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:55.062003 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:55.062007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:55.062007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:55.062019 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:55.064150 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:55.064635 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:55.064638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:55.064638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:55.064771 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:55.065048 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:55.065049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:55.065050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:55.065366 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:55.065927 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:55.066459 (+ 532us) server_negotiation.cc:299] Negotiation successful
0812 14:26:55.066568 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260812 14:26:55.067960 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:26:55.076804 30977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:55.069561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:38183 (local address 127.0.0.1:45193)
0812 14:26:55.069596 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:55.069597 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:55.069609 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:26:55.069631 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:55.069631 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:55.069642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:55.069707 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:55.070299 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:55.071162 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:55.072143 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:55.072486 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:55.072560 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:55.073918 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:55.073920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:55.073920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:55.073927 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:55.075536 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:55.075951 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:55.075952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:55.075953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:55.076025 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:55.076285 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:55.076286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:55.076286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:55.076418 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:55.076537 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:55.076644 (+ 107us) server_negotiation.cc:299] Negotiation successful
0812 14:26:55.076708 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:26:55.079053 30997 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:55.079509 30997 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:55.082625 30997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e90d7c924434bf694b098ea773f2b7e: Generated new TSK 0
I20260812 14:26:55.082799 30997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:55.088155 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 30894
W20260812 14:26:55.096236 30872 connection.cc:535] client connection to 127.0.0.1:45193 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45193): Connection reset by peer (error 104)
2026-08-12T14:26:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:55 dist-test-slave-8w9v krb5kdc[31006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:55 dist-test-slave-8w9v krb5kdc[31006](info): set up 2 sockets
Aug 12 14:26:55 dist-test-slave-8w9v krb5kdc[31006](info): commencing operation
krb5kdc: starting...
W20260812 14:26:56.207274 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:56 dist-test-slave-8w9v krb5kdc[31006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:26:56.335119 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39081
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:56.419242 31021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:56.419508 31021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:56.419548 31021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:56.420750 31021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:56.420828 31021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:56.420853 31021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:56.420883 31021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:56.420914 31021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:56.422431 31021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39081
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:56.422719 31021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:56.422969 31021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:56.428030 31028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:56.428069 31021 server_base.cc:1034] running on GCE node
W20260812 14:26:56.428035 31027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:56.428030 31031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:56.428500 31021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:56.428817 31021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:56.429991 31021 hybrid_clock.cc:648] HybridClock initialized: now 1786544816429983 us; error 35 us; skew 500 ppm
Aug 12 14:26:56 dist-test-slave-8w9v krb5kdc[31006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:56.443584 31021 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:26:56.444525 31021 webserver.cc:460] Webserver started at http://127.0.0.1:33149/ using document root <none> and password file <none>
I20260812 14:26:56.445014 31021 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:56.445156 31021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:56.445472 31021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:56.447505 31021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "f7914d76aff1430893cc7b48b07e96c8"
format_stamp: "Formatted at 2026-08-12 14:26:56 on dist-test-slave-8w9v"
server_key: "b07f9b46034c13a254244f311edea3e2"
server_key_iv: "b17f5d88a364570447fea47f89088209"
server_key_version: "encryptionkey@0"
I20260812 14:26:56.448285 31021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "f7914d76aff1430893cc7b48b07e96c8"
format_stamp: "Formatted at 2026-08-12 14:26:56 on dist-test-slave-8w9v"
server_key: "b07f9b46034c13a254244f311edea3e2"
server_key_iv: "b17f5d88a364570447fea47f89088209"
server_key_version: "encryptionkey@0"
I20260812 14:26:56.451138 31021 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 14:26:56.452517 31038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:56.452808 31021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:56.452989 31021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "f7914d76aff1430893cc7b48b07e96c8"
format_stamp: "Formatted at 2026-08-12 14:26:56 on dist-test-slave-8w9v"
server_key: "b07f9b46034c13a254244f311edea3e2"
server_key_iv: "b17f5d88a364570447fea47f89088209"
server_key_version: "encryptionkey@0"
I20260812 14:26:56.453169 31021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:56.478047 31021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:56.481890 31021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:56.482205 31021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:56.490047 31021 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39081
I20260812 14:26:56.490156 31101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39081 every 8 connection(s)
I20260812 14:26:56.490863 31021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:56.491681 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 31021
I20260812 14:26:56.491899 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:56.492317 5712 external_mini_cluster.cc:1428] Setting key 9a55b16c296639887e0e651b34f489c8
I20260812 14:26:56.492766 31102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:56.499558 31102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8: Bootstrap starting.
I20260812 14:26:56.500967 31102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:56.501632 31102 log.cc:824] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:56.503175 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:56.493276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:44771 (local address 127.0.0.1:39081)
0812 14:26:56.493529 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:56.493538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:56.493583 (+ 45us) server_negotiation.cc:407] Connection header received
0812 14:26:56.494788 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:56.494805 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:56.494904 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:56.495111 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:56.495625 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:56.497317 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:56.499396 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:56.500230 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:56.500401 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:56.501720 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:56.501732 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:56.501732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:56.502093 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:56.502146 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:56.502175 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:56.502301 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:56.502591 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 14:26:56.502655 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":123,"threads_started":1}
I20260812 14:26:56.503279 31102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8: No bootstrap required, opened a new log
I20260812 14:26:56.505038 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:26:56.507508 31102 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7914d76aff1430893cc7b48b07e96c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39081 } }
I20260812 14:26:56.508054 31102 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:56.508271 31102 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7914d76aff1430893cc7b48b07e96c8, State: Initialized, Role: FOLLOWER
I20260812 14:26:56.508665 31102 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7914d76aff1430893cc7b48b07e96c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39081 } }
I20260812 14:26:56.508935 31102 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:56.509059 31102 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:56.509179 31102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:56.510607 31102 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7914d76aff1430893cc7b48b07e96c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39081 } }
I20260812 14:26:56.510859 31102 leader_election.cc:302] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7914d76aff1430893cc7b48b07e96c8; no voters:
I20260812 14:26:56.511216 31102 leader_election.cc:288] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:56.511268 31113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:56.511686 31113 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [term 1 LEADER]: Becoming Leader. State: Replica: f7914d76aff1430893cc7b48b07e96c8, State: Running, Role: LEADER
I20260812 14:26:56.511842 31102 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:56.512072 31113 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7914d76aff1430893cc7b48b07e96c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39081 } }
I20260812 14:26:56.512807 31113 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7914d76aff1430893cc7b48b07e96c8. Latest consensus state: current_term: 1 leader_uuid: "f7914d76aff1430893cc7b48b07e96c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7914d76aff1430893cc7b48b07e96c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39081 } } }
I20260812 14:26:56.513298 31113 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:56.514170 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:56.507387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:59329 (local address 127.0.0.1:39081)
0812 14:26:56.507489 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:56.507491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:56.507649 (+ 158us) server_negotiation.cc:407] Connection header received
0812 14:26:56.507701 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:56.507702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:56.507725 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:56.507911 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:56.508250 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:56.509685 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:56.511165 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:56.511761 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:56.511945 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:56.513561 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:56.513563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:56.513563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:56.513577 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:56.513606 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:56.513614 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:56.513725 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:56.514009 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 14:26:56.514063 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 14:26:56.514972 31115 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7914d76aff1430893cc7b48b07e96c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7914d76aff1430893cc7b48b07e96c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39081 } } }
I20260812 14:26:56.515609 31115 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8 [sys.catalog]: This master's current role is: LEADER
W20260812 14:26:56.516881 31135 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:26:56.516952 31135 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:26:56.517113 31125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:56.517369 31125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:56.520002 31125 catalog_manager.cc:1347] Generated new cluster ID: e8d258b38ec94ab69c65046bdb3a29be
I20260812 14:26:56.520102 31125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:56.537133 31125 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:56.538470 31125 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:56.550118 31125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7914d76aff1430893cc7b48b07e96c8: Generated new TSK 0
I20260812 14:26:56.550484 31125 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:56.559047 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 31021
W20260812 14:26:56.574553 31004 connection.cc:535] client connection to 127.0.0.1:39081 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39081): Connection reset by peer (error 104)
2026-08-12T14:26:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1481 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:56 dist-test-slave-8w9v krb5kdc[31142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:56 dist-test-slave-8w9v krb5kdc[31142](info): set up 2 sockets
Aug 12 14:26:56 dist-test-slave-8w9v krb5kdc[31142](info): commencing operation
krb5kdc: starting...
W20260812 14:26:57.748914 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.002s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:57 dist-test-slave-8w9v krb5kdc[31142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:26:57.887017 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:57.979698 31156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:57.980026 31156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:57.980069 31156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:57.982086 31156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:57.982147 31156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:57.982167 31156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:57.982187 31156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:57.982265 31156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:57.984614 31156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34173
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:57.984937 31156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:57.985234 31156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:57.991168 31162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:57.991164 31164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:57.991168 31161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:57.992132 31156 server_base.cc:1034] running on GCE node
I20260812 14:26:57.992477 31156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:57.992889 31156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:57.994064 31156 hybrid_clock.cc:648] HybridClock initialized: now 1786544817994053 us; error 29 us; skew 500 ppm
Aug 12 14:26:57 dist-test-slave-8w9v krb5kdc[31142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:57.996935 31156 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:26:57.997408 31156 webserver.cc:460] Webserver started at http://127.0.0.1:39327/ using document root <none> and password file <none>
I20260812 14:26:57.997618 31156 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:57.997659 31156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:57.997752 31156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:57.998783 31156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "9d2d4dc761524d4899d511f96798969f"
format_stamp: "Formatted at 2026-08-12 14:26:57 on dist-test-slave-8w9v"
server_key: "b1b2952d54701347aee1d02562ef1aa3"
server_key_iv: "c50754ad474f0462fa2f5288777c02ee"
server_key_version: "encryptionkey@0"
I20260812 14:26:57.999168 31156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "9d2d4dc761524d4899d511f96798969f"
format_stamp: "Formatted at 2026-08-12 14:26:57 on dist-test-slave-8w9v"
server_key: "b1b2952d54701347aee1d02562ef1aa3"
server_key_iv: "c50754ad474f0462fa2f5288777c02ee"
server_key_version: "encryptionkey@0"
I20260812 14:26:58.000765 31156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:26:58.001700 31173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:58.001888 31156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:26:58.002009 31156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "9d2d4dc761524d4899d511f96798969f"
format_stamp: "Formatted at 2026-08-12 14:26:57 on dist-test-slave-8w9v"
server_key: "b1b2952d54701347aee1d02562ef1aa3"
server_key_iv: "c50754ad474f0462fa2f5288777c02ee"
server_key_version: "encryptionkey@0"
I20260812 14:26:58.002110 31156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:58.006168 31156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:58.008687 31156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:58.008852 31156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:58.013124 31156 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34173
I20260812 14:26:58.013156 31233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34173 every 8 connection(s)
I20260812 14:26:58.013515 31156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:58.014508 31234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:58.016862 31234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f: Bootstrap starting.
I20260812 14:26:58.017449 31234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:58.017696 31234 log.cc:824] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:58.018354 31234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f: No bootstrap required, opened a new log
I20260812 14:26:58.019919 31234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2d4dc761524d4899d511f96798969f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34173 } }
I20260812 14:26:58.020058 31234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:58.020080 31234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2d4dc761524d4899d511f96798969f, State: Initialized, Role: FOLLOWER
I20260812 14:26:58.020277 31234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2d4dc761524d4899d511f96798969f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34173 } }
I20260812 14:26:58.020344 31234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:58.020367 31234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:58.020395 31234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:58.020902 31234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2d4dc761524d4899d511f96798969f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34173 } }
I20260812 14:26:58.020988 31234 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d2d4dc761524d4899d511f96798969f; no voters:
I20260812 14:26:58.021114 31234 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:58.021167 31237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:58.021332 31237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [term 1 LEADER]: Becoming Leader. State: Replica: 9d2d4dc761524d4899d511f96798969f, State: Running, Role: LEADER
I20260812 14:26:58.021413 31234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:58.021489 31237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2d4dc761524d4899d511f96798969f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34173 } }
I20260812 14:26:58.021945 31239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2d4dc761524d4899d511f96798969f. Latest consensus state: current_term: 1 leader_uuid: "9d2d4dc761524d4899d511f96798969f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2d4dc761524d4899d511f96798969f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34173 } } }
I20260812 14:26:58.022099 31239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:58.022084 31238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d2d4dc761524d4899d511f96798969f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2d4dc761524d4899d511f96798969f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34173 } } }
I20260812 14:26:58.022393 31238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:58.022576 31247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:58.022583 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 31156
I20260812 14:26:58.022727 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:58.022948 5712 external_mini_cluster.cc:1428] Setting key 9b98bf077e5a396d84cbfa0f48c53089
I20260812 14:26:58.023146 31247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:58.024864 31247 catalog_manager.cc:1347] Generated new cluster ID: 6ae304ab999842a2a80197bcf8a9c3a5
I20260812 14:26:58.024983 31247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:26:58.025851 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:58.023621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:44637 (local address 127.0.0.1:34173)
0812 14:26:58.023880 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:58.023897 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:58.023930 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:26:58.024049 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:58.024056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:58.024086 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:58.024265 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:58.025380 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:58.025384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:58.025384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:58.025389 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:58.025437 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:58.025450 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:58.025515 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:58.025610 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 14:26:58.025653 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":58,"threads_started":1}
I20260812 14:26:58.026746 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:26:58.029490 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:58.028374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:52765 (local address 127.0.0.1:34173)
0812 14:26:58.028424 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:58.028426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:58.028553 (+ 127us) server_negotiation.cc:407] Connection header received
0812 14:26:58.028604 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:58.028605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:58.028611 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:58.028670 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:58.029262 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:58.029263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:58.029263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:26:58.029268 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:58.029284 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:26:58.029290 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:58.029353 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:58.029414 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 14:26:58.029427 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:26:58.044467 31247 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:58.044694 31247 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:58.051501 31247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d2d4dc761524d4899d511f96798969f: Generated new TSK 0
I20260812 14:26:58.051724 31247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:58.056423 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 31156
2026-08-12T14:26:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1485 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:58 dist-test-slave-8w9v krb5kdc[31271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:58 dist-test-slave-8w9v krb5kdc[31271](info): set up 2 sockets
Aug 12 14:26:58 dist-test-slave-8w9v krb5kdc[31271](info): commencing operation
krb5kdc: starting...
W20260812 14:26:59.184015 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:26:59 dist-test-slave-8w9v krb5kdc[31271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:26:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:26:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:26:59.315330 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:35261
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:35261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:26:59.398743 31287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:26:59.398967 31287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:26:59.398990 31287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:26:59.400068 31287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:26:59.400104 31287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:26:59.400193 31287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:26:59.400228 31287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:26:59.400241 31287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:26:59.401717 31287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:35261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:35261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:26:59.401925 31287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:26:59.402097 31287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:26:59.407572 31297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:59.407647 31287 server_base.cc:1034] running on GCE node
W20260812 14:26:59.407567 31293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:26:59.407577 31292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:26:59.408178 31287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:26:59.408519 31287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:26:59.409667 31287 hybrid_clock.cc:648] HybridClock initialized: now 1786544819409658 us; error 32 us; skew 500 ppm
Aug 12 14:26:59 dist-test-slave-8w9v krb5kdc[31271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544819, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:26:59.412217 31287 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:26:59.412725 31287 webserver.cc:460] Webserver started at http://127.5.148.62:36075/ using document root <none> and password file <none>
I20260812 14:26:59.412979 31287 fs_manager.cc:359] Metadata directory not provided
I20260812 14:26:59.413053 31287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:26:59.413203 31287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:26:59.414299 31287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "c6c1ac6a55a9466b8e14ebc221fb0e83"
format_stamp: "Formatted at 2026-08-12 14:26:59 on dist-test-slave-8w9v"
server_key: "218ad481d45769bd2181459414453a77"
server_key_iv: "9a1277afb07b752bf8197d8517841376"
server_key_version: "encryptionkey@0"
I20260812 14:26:59.414697 31287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "c6c1ac6a55a9466b8e14ebc221fb0e83"
format_stamp: "Formatted at 2026-08-12 14:26:59 on dist-test-slave-8w9v"
server_key: "218ad481d45769bd2181459414453a77"
server_key_iv: "9a1277afb07b752bf8197d8517841376"
server_key_version: "encryptionkey@0"
I20260812 14:26:59.416190 31287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:26:59.416993 31305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:26:59.417173 31287 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:26:59.417285 31287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "c6c1ac6a55a9466b8e14ebc221fb0e83"
format_stamp: "Formatted at 2026-08-12 14:26:59 on dist-test-slave-8w9v"
server_key: "218ad481d45769bd2181459414453a77"
server_key_iv: "9a1277afb07b752bf8197d8517841376"
server_key_version: "encryptionkey@0"
I20260812 14:26:59.417366 31287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:26:59.438922 31287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:26:59.441035 31287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:26:59.441211 31287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:26:59.445631 31287 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:35261
I20260812 14:26:59.445667 31369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:35261 every 8 connection(s)
I20260812 14:26:59.446026 31287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:26:59.447000 31370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:26:59.449757 31370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83: Bootstrap starting.
I20260812 14:26:59.450323 31370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83: Neither blocks nor log segments found. Creating new log.
I20260812 14:26:59.450562 31370 log.cc:824] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83: Log is configured to *not* fsync() on all Append() calls
I20260812 14:26:59.451261 31370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83: No bootstrap required, opened a new log
I20260812 14:26:59.451483 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 31287
I20260812 14:26:59.451603 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:26:59.451815 5712 external_mini_cluster.cc:1428] Setting key 0ba0feabfe7d43970bab6fbe3e6f105d
I20260812 14:26:59.453480 31370 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c1ac6a55a9466b8e14ebc221fb0e83" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35261 } }
I20260812 14:26:59.453653 31370 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:26:59.453698 31370 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c1ac6a55a9466b8e14ebc221fb0e83, State: Initialized, Role: FOLLOWER
I20260812 14:26:59.453907 31370 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c1ac6a55a9466b8e14ebc221fb0e83" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35261 } }
I20260812 14:26:59.454010 31370 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:26:59.454046 31370 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:26:59.454102 31370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:26:59.455158 31370 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c1ac6a55a9466b8e14ebc221fb0e83" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35261 } }
I20260812 14:26:59.455286 31370 leader_election.cc:302] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6c1ac6a55a9466b8e14ebc221fb0e83; no voters:
I20260812 14:26:59.455468 31370 leader_election.cc:288] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:26:59.455505 31376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:26:59.455701 31376 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [term 1 LEADER]: Becoming Leader. State: Replica: c6c1ac6a55a9466b8e14ebc221fb0e83, State: Running, Role: LEADER
I20260812 14:26:59.455737 31370 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:26:59.455874 31376 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c1ac6a55a9466b8e14ebc221fb0e83" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35261 } }
I20260812 14:26:59.456650 31379 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6c1ac6a55a9466b8e14ebc221fb0e83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c1ac6a55a9466b8e14ebc221fb0e83" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35261 } } }
I20260812 14:26:59.456663 31378 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6c1ac6a55a9466b8e14ebc221fb0e83. Latest consensus state: current_term: 1 leader_uuid: "c6c1ac6a55a9466b8e14ebc221fb0e83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c1ac6a55a9466b8e14ebc221fb0e83" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 35261 } } }
I20260812 14:26:59.456796 31379 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:59.456816 31378 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83 [sys.catalog]: This master's current role is: LEADER
I20260812 14:26:59.457041 31388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:26:59.457666 31388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:26:59.459674 31388 catalog_manager.cc:1347] Generated new cluster ID: dcd008e75e6543c9ae53fee926229d27
I20260812 14:26:59.459726 31388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:26:59 dist-test-slave-8w9v krb5kdc[31271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:26:59.464294 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:59.452379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57345 (local address 127.5.148.62:35261)
0812 14:26:59.452577 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:59.452586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:59.452623 (+ 37us) server_negotiation.cc:407] Connection header received
0812 14:26:59.453461 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:59.453467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:59.453514 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:59.453629 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:59.454486 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:59.455478 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:59.456550 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:59.457310 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:59.457458 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:59.460383 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:59.460387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:59.460387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:59.460400 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:59.462503 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:59.462949 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:59.462952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:59.462952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:59.463091 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:59.463330 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:59.463331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:59.463331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:59.463658 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:59.463824 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:59.463942 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 14:26:59.464056 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":115,"threads_started":1}
I20260812 14:26:59.465566 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:26:59.476063 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:26:59.468558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42647 (local address 127.5.148.62:35261)
0812 14:26:59.468600 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:26:59.468602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:26:59.468772 (+ 170us) server_negotiation.cc:407] Connection header received
0812 14:26:59.468818 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:26:59.468819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:26:59.468829 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:26:59.468876 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:26:59.469546 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:59.470314 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:59.471350 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:26:59.471672 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:26:59.471738 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:26:59.473207 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:26:59.473209 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:26:59.473209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:26:59.473216 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:26:59.474793 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:59.475223 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:59.475224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:59.475225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:59.475280 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:26:59.475514 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:26:59.475514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:26:59.475515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:26:59.475634 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:26:59.475736 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 14:26:59.475891 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 14:26:59.475961 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:26:59.482535 31388 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:26:59.483279 31388 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:26:59.490434 31388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6c1ac6a55a9466b8e14ebc221fb0e83: Generated new TSK 0
I20260812 14:26:59.490651 31388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:26:59.499850 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 31287
W20260812 14:26:59.510425 31269 connection.cc:535] client connection to 127.5.148.62:35261 recv error: Network error: failed to read from TLS socket (remote: 127.5.148.62:35261): Connection reset by peer (error 104)
2026-08-12T14:26:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:26:59 dist-test-slave-8w9v krb5kdc[31410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:26:59 dist-test-slave-8w9v krb5kdc[31410](info): set up 2 sockets
Aug 12 14:26:59 dist-test-slave-8w9v krb5kdc[31410](info): commencing operation
krb5kdc: starting...
W20260812 14:27:00.672430 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:27:00 dist-test-slave-8w9v krb5kdc[31410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:27:00.799216 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:42987
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:42987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:00.883045 31425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:00.883409 31425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:00.883514 31425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:00.884770 31425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:00.884842 31425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:00.884871 31425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:00.884891 31425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:00.884909 31425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:00.886487 31425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:42987
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:42987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:00.886783 31425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:00.887022 31425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:00.892138 31435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:00.892181 31437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:00.892206 31425 server_base.cc:1034] running on GCE node
W20260812 14:27:00.892218 31434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:00.892606 31425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:00.892951 31425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:00.894124 31425 hybrid_clock.cc:648] HybridClock initialized: now 1786544820894101 us; error 35 us; skew 500 ppm
Aug 12 14:27:00 dist-test-slave-8w9v krb5kdc[31410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544820, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:00.896644 31425 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:27:00.897156 31425 webserver.cc:460] Webserver started at http://127.5.148.62:38229/ using document root <none> and password file <none>
I20260812 14:27:00.897396 31425 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:00.897468 31425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:00.897606 31425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:00.898707 31425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "62ee562a897a4927bbb95944d27ae3c0"
format_stamp: "Formatted at 2026-08-12 14:27:00 on dist-test-slave-8w9v"
server_key: "20502ea84880266dd7f6ff7865eae1ac"
server_key_iv: "9d5f4e5c65fe8f799f4afc852506aee5"
server_key_version: "encryptionkey@0"
I20260812 14:27:00.899098 31425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "62ee562a897a4927bbb95944d27ae3c0"
format_stamp: "Formatted at 2026-08-12 14:27:00 on dist-test-slave-8w9v"
server_key: "20502ea84880266dd7f6ff7865eae1ac"
server_key_iv: "9d5f4e5c65fe8f799f4afc852506aee5"
server_key_version: "encryptionkey@0"
I20260812 14:27:00.900534 31425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:00.901291 31444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:00.901485 31425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:00.901593 31425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "62ee562a897a4927bbb95944d27ae3c0"
format_stamp: "Formatted at 2026-08-12 14:27:00 on dist-test-slave-8w9v"
server_key: "20502ea84880266dd7f6ff7865eae1ac"
server_key_iv: "9d5f4e5c65fe8f799f4afc852506aee5"
server_key_version: "encryptionkey@0"
I20260812 14:27:00.901669 31425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:00.921113 31425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:00.923151 31425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:00.923336 31425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:00.927752 31425 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:42987
I20260812 14:27:00.927791 31503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:42987 every 8 connection(s)
I20260812 14:27:00.928292 31425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:00.929217 31504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:00.931545 31504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0: Bootstrap starting.
I20260812 14:27:00.932134 31504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:00.932379 31504 log.cc:824] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:00.933071 31504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0: No bootstrap required, opened a new log
I20260812 14:27:00.934291 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 31425
I20260812 14:27:00.934423 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:27:00.934631 5712 external_mini_cluster.cc:1428] Setting key 0a7a048262aa0c47fddcd5524fc0cb86
I20260812 14:27:00.935091 31504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ee562a897a4927bbb95944d27ae3c0" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 42987 } }
I20260812 14:27:00.935288 31504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:00.935395 31504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ee562a897a4927bbb95944d27ae3c0, State: Initialized, Role: FOLLOWER
I20260812 14:27:00.935577 31504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ee562a897a4927bbb95944d27ae3c0" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 42987 } }
I20260812 14:27:00.935664 31504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:00.935696 31504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:00.935748 31504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:00.936990 31504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ee562a897a4927bbb95944d27ae3c0" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 42987 } }
I20260812 14:27:00.937106 31504 leader_election.cc:302] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62ee562a897a4927bbb95944d27ae3c0; no voters:
I20260812 14:27:00.937258 31504 leader_election.cc:288] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:00.937311 31510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:00.937491 31504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:00.937510 31510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [term 1 LEADER]: Becoming Leader. State: Replica: 62ee562a897a4927bbb95944d27ae3c0, State: Running, Role: LEADER
I20260812 14:27:00.937734 31510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ee562a897a4927bbb95944d27ae3c0" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 42987 } }
I20260812 14:27:00.938122 31511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62ee562a897a4927bbb95944d27ae3c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ee562a897a4927bbb95944d27ae3c0" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 42987 } } }
I20260812 14:27:00.938210 31511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:00.938369 31512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ee562a897a4927bbb95944d27ae3c0. Latest consensus state: current_term: 1 leader_uuid: "62ee562a897a4927bbb95944d27ae3c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ee562a897a4927bbb95944d27ae3c0" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 42987 } } }
I20260812 14:27:00.938452 31512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:00.938885 31520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:00.939464 31520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:00.941171 31509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:00.935194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57737 (local address 127.5.148.62:42987)
0812 14:27:00.935445 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:00.935456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:00.935487 (+ 31us) server_negotiation.cc:407] Connection header received
0812 14:27:00.935604 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:00.935611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:00.935668 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:00.935815 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:00.936046 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:00.937564 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:00.938811 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:00.939386 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:00.939532 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:00.940330 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:00.940335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:00.940335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:00.940582 (+ 247us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:00.940626 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:00.940642 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:00.940733 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:00.940928 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 14:27:00.940957 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":107,"threads_started":1}
I20260812 14:27:00.941201 31520 catalog_manager.cc:1347] Generated new cluster ID: 518056ee9080418ea40b36d89cbe1902
I20260812 14:27:00.941237 31520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:00.942175 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:27:00.950835 31520 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:00.951562 31520 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:00.954531 31509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:00.950646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42767 (local address 127.5.148.62:42987)
0812 14:27:00.950713 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:00.950716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:00.951006 (+ 290us) server_negotiation.cc:407] Connection header received
0812 14:27:00.951036 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:00.951037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:00.951049 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:00.951118 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:00.951301 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:00.952042 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:00.953123 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:00.953413 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:00.953476 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:00.954253 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:00.954254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:00.954254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:00.954259 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:00.954275 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:00.954279 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:00.954335 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:00.954441 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 14:27:00.954454 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":39}
I20260812 14:27:00.955770 31520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62ee562a897a4927bbb95944d27ae3c0: Generated new TSK 0
I20260812 14:27:00.955945 31520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:27:00.961295 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 31425
W20260812 14:27:00.972924 31408 connection.cc:535] client connection to 127.5.148.62:42987 recv error: Network error: failed to read from TLS socket (remote: 127.5.148.62:42987): Connection reset by peer (error 104)
2026-08-12T14:27:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1462 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:01 dist-test-slave-8w9v krb5kdc[31538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:01 dist-test-slave-8w9v krb5kdc[31538](info): set up 2 sockets
Aug 12 14:27:01 dist-test-slave-8w9v krb5kdc[31538](info): commencing operation
krb5kdc: starting...
W20260812 14:27:02.089061 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:27:02 dist-test-slave-8w9v krb5kdc[31538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:27:02.218515 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:37933
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:37933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:02.302071 31552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:02.302313 31552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:02.302353 31552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:02.303517 31552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:02.303591 31552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:02.303614 31552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:02.303637 31552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:02.303668 31552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:02.305176 31552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:37933
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:37933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:02.305466 31552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:02.305696 31552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:02.310837 31564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:02.310768 31561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:02.310772 31560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:02.310976 31552 server_base.cc:1034] running on GCE node
I20260812 14:27:02.311359 31552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:02.311630 31552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:02.312773 31552 hybrid_clock.cc:648] HybridClock initialized: now 1786544822312759 us; error 30 us; skew 500 ppm
Aug 12 14:27:02 dist-test-slave-8w9v krb5kdc[31538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544822, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:02.320464 31552 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:27:02.320955 31552 webserver.cc:460] Webserver started at http://127.5.148.62:35311/ using document root <none> and password file <none>
I20260812 14:27:02.321228 31552 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:02.321300 31552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:02.321455 31552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:02.322628 31552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "fd1c252c8606463aa0e35dc73bb0f086"
format_stamp: "Formatted at 2026-08-12 14:27:02 on dist-test-slave-8w9v"
server_key: "f13f35b7a187590c041a3a84eb321fde"
server_key_iv: "860f6e827fa628d2cfa3ae0e294b4ef2"
server_key_version: "encryptionkey@0"
I20260812 14:27:02.323028 31552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "fd1c252c8606463aa0e35dc73bb0f086"
format_stamp: "Formatted at 2026-08-12 14:27:02 on dist-test-slave-8w9v"
server_key: "f13f35b7a187590c041a3a84eb321fde"
server_key_iv: "860f6e827fa628d2cfa3ae0e294b4ef2"
server_key_version: "encryptionkey@0"
I20260812 14:27:02.324554 31552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:02.325349 31572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:02.325546 31552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:02.325632 31552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "fd1c252c8606463aa0e35dc73bb0f086"
format_stamp: "Formatted at 2026-08-12 14:27:02 on dist-test-slave-8w9v"
server_key: "f13f35b7a187590c041a3a84eb321fde"
server_key_iv: "860f6e827fa628d2cfa3ae0e294b4ef2"
server_key_version: "encryptionkey@0"
I20260812 14:27:02.325690 31552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:02.329073 31552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:02.331007 31552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:02.331113 31552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:02.335103 31552 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:37933
I20260812 14:27:02.335175 31632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:37933 every 8 connection(s)
I20260812 14:27:02.335497 31552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:02.336692 31633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:02.339003 31633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086: Bootstrap starting.
I20260812 14:27:02.339599 31633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:02.339920 31633 log.cc:824] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:02.340672 31633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086: No bootstrap required, opened a new log
I20260812 14:27:02.342300 31633 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1c252c8606463aa0e35dc73bb0f086" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37933 } }
I20260812 14:27:02.342469 31633 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:02.342523 31633 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1c252c8606463aa0e35dc73bb0f086, State: Initialized, Role: FOLLOWER
I20260812 14:27:02.342693 31633 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1c252c8606463aa0e35dc73bb0f086" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37933 } }
I20260812 14:27:02.342800 31633 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:02.342846 31633 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:02.342901 31633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:02.343758 31633 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1c252c8606463aa0e35dc73bb0f086" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37933 } }
I20260812 14:27:02.343885 31633 leader_election.cc:302] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd1c252c8606463aa0e35dc73bb0f086; no voters:
I20260812 14:27:02.344066 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 31552
I20260812 14:27:02.344108 31633 leader_election.cc:288] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:02.344172 31636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:02.344226 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:27:02.344300 31636 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [term 1 LEADER]: Becoming Leader. State: Replica: fd1c252c8606463aa0e35dc73bb0f086, State: Running, Role: LEADER
I20260812 14:27:02.344399 31633 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:02.344434 5712 external_mini_cluster.cc:1428] Setting key db151f9d8bad73262e3010aec11835f4
I20260812 14:27:02.344460 31636 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1c252c8606463aa0e35dc73bb0f086" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37933 } }
I20260812 14:27:02.344871 31638 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd1c252c8606463aa0e35dc73bb0f086. Latest consensus state: current_term: 1 leader_uuid: "fd1c252c8606463aa0e35dc73bb0f086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1c252c8606463aa0e35dc73bb0f086" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37933 } } }
I20260812 14:27:02.344875 31637 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd1c252c8606463aa0e35dc73bb0f086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1c252c8606463aa0e35dc73bb0f086" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37933 } } }
I20260812 14:27:02.345016 31638 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:02.345054 31637 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086 [sys.catalog]: This master's current role is: LEADER
W20260812 14:27:02.346436 31657 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:27:02.346882 31657 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:27:02.347009 31649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:02.347222 31649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:02.347651 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:02.345075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38541 (local address 127.5.148.62:37933)
0812 14:27:02.345805 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:02.345815 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:02.345854 (+ 39us) server_negotiation.cc:407] Connection header received
0812 14:27:02.345952 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:02.345957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:02.345982 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:02.346137 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:02.346987 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:02.346991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:02.346991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:02.346996 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:02.347044 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:02.347060 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:02.347357 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:02.347390 (+ 33us) server_negotiation.cc:299] Negotiation successful
0812 14:27:02.347455 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":65,"threads_started":1}
I20260812 14:27:02.348855 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:27:02.348908 31649 catalog_manager.cc:1347] Generated new cluster ID: e3dc10f5c8a94a2b95fecda1fe196a4a
I20260812 14:27:02.348953 31649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:02.351567 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:02.350460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41849 (local address 127.5.148.62:37933)
0812 14:27:02.350495 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:02.350497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:02.350610 (+ 113us) server_negotiation.cc:407] Connection header received
0812 14:27:02.350641 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:02.350642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:02.350648 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:02.350721 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:02.351331 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:02.351332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:02.351332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:02.351336 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:02.351353 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:02.351357 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:02.351430 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:02.351482 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 14:27:02.351495 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:27:02.370710 31649 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:02.370934 31649 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:02.383327 31649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd1c252c8606463aa0e35dc73bb0f086: Generated new TSK 0
I20260812 14:27:02.383554 31649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:27:02.389707 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 31552
2026-08-12T14:27:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:02 dist-test-slave-8w9v krb5kdc[31670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:02 dist-test-slave-8w9v krb5kdc[31670](info): set up 2 sockets
Aug 12 14:27:02 dist-test-slave-8w9v krb5kdc[31670](info): commencing operation
krb5kdc: starting...
W20260812 14:27:03.516845 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:27:03 dist-test-slave-8w9v krb5kdc[31670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:27:03.646533 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:40383
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:40383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:03.729993 31685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:03.730208 31685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:03.730232 31685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:03.731345 31685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:03.731383 31685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:03.731397 31685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:03.731410 31685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:03.731431 31685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:03.732883 31685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:40383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:40383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:03.733155 31685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:03.733392 31685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:03.738554 31694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:03.738588 31691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:03.738624 31690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:03.738668 31685 server_base.cc:1034] running on GCE node
I20260812 14:27:03.739035 31685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:03.739401 31685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:03.740578 31685 hybrid_clock.cc:648] HybridClock initialized: now 1786544823740569 us; error 36 us; skew 500 ppm
Aug 12 14:27:03 dist-test-slave-8w9v krb5kdc[31670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544823, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:03.743218 31685 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:27:03.743718 31685 webserver.cc:460] Webserver started at http://127.5.148.62:40277/ using document root <none> and password file <none>
I20260812 14:27:03.743976 31685 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:03.744024 31685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:03.744186 31685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:03.745926 31685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "a8858c00785e418ead2c5908393c0b9c"
format_stamp: "Formatted at 2026-08-12 14:27:03 on dist-test-slave-8w9v"
server_key: "db56c55ec7706c9bcb04f7f04413ac50"
server_key_iv: "6c24cb554f607e776675f5beed162c42"
server_key_version: "encryptionkey@0"
I20260812 14:27:03.746462 31685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "a8858c00785e418ead2c5908393c0b9c"
format_stamp: "Formatted at 2026-08-12 14:27:03 on dist-test-slave-8w9v"
server_key: "db56c55ec7706c9bcb04f7f04413ac50"
server_key_iv: "6c24cb554f607e776675f5beed162c42"
server_key_version: "encryptionkey@0"
I20260812 14:27:03.748627 31685 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260812 14:27:03.750595 31702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:03.750787 31685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260812 14:27:03.750913 31685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "a8858c00785e418ead2c5908393c0b9c"
format_stamp: "Formatted at 2026-08-12 14:27:03 on dist-test-slave-8w9v"
server_key: "db56c55ec7706c9bcb04f7f04413ac50"
server_key_iv: "6c24cb554f607e776675f5beed162c42"
server_key_version: "encryptionkey@0"
I20260812 14:27:03.750993 31685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:03.764395 31685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:03.766265 31685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:03.766426 31685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:03.770607 31685 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:40383
I20260812 14:27:03.770671 31765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:40383 every 8 connection(s)
I20260812 14:27:03.771037 31685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:03.771986 31766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:03.772333 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 31685
I20260812 14:27:03.772459 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:27:03.772657 5712 external_mini_cluster.cc:1428] Setting key f17cef74ed5a46b1e12eddda6e39867a
I20260812 14:27:03.775218 31766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c: Bootstrap starting.
I20260812 14:27:03.775847 31766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:03.776233 31766 log.cc:824] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:03.776948 31766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c: No bootstrap required, opened a new log
I20260812 14:27:03.779176 31766 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8858c00785e418ead2c5908393c0b9c" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40383 } }
I20260812 14:27:03.779358 31766 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:03.779394 31766 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8858c00785e418ead2c5908393c0b9c, State: Initialized, Role: FOLLOWER
Aug 12 14:27:03 dist-test-slave-8w9v krb5kdc[31670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:27:03.779534 31766 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8858c00785e418ead2c5908393c0b9c" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40383 } }
I20260812 14:27:03.779640 31766 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:03.779687 31766 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:03.779762 31766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:03.780581 31766 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8858c00785e418ead2c5908393c0b9c" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40383 } }
I20260812 14:27:03.780706 31766 leader_election.cc:302] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8858c00785e418ead2c5908393c0b9c; no voters:
I20260812 14:27:03.780893 31766 leader_election.cc:288] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:03.780961 31771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:03.781168 31771 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [term 1 LEADER]: Becoming Leader. State: Replica: a8858c00785e418ead2c5908393c0b9c, State: Running, Role: LEADER
I20260812 14:27:03.781167 31766 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:03.781378 31771 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8858c00785e418ead2c5908393c0b9c" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40383 } }
I20260812 14:27:03.781800 31773 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8858c00785e418ead2c5908393c0b9c. Latest consensus state: current_term: 1 leader_uuid: "a8858c00785e418ead2c5908393c0b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8858c00785e418ead2c5908393c0b9c" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40383 } } }
I20260812 14:27:03.781930 31773 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:03.782541 31772 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8858c00785e418ead2c5908393c0b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8858c00785e418ead2c5908393c0b9c" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 40383 } } }
I20260812 14:27:03.782630 31772 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c [sys.catalog]: This master's current role is: LEADER
W20260812 14:27:03.783131 31788 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:27:03.783205 31788 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:27:03.783298 31780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:03.783474 31780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:03.784376 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:03.773251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41223 (local address 127.5.148.62:40383)
0812 14:27:03.773453 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:03.773461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:03.773487 (+ 26us) server_negotiation.cc:407] Connection header received
0812 14:27:03.773903 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:03.773910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:03.773954 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:03.774073 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:03.774913 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:03.775857 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:03.776776 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:03.777235 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:03.777353 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:03.780039 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:03.780043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:03.780043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:03.780056 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:03.782164 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:03.782615 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:03.782618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:03.782618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:03.782771 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:03.783083 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:03.783084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:03.783084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:03.783384 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:03.783542 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:03.783998 (+ 456us) server_negotiation.cc:299] Negotiation successful
0812 14:27:03.784177 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260812 14:27:03.785022 31780 catalog_manager.cc:1347] Generated new cluster ID: fbbf3fe73dba4acc8a3d07549c71d4fa
I20260812 14:27:03.785073 31780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:03.785359 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:27:03.794535 31780 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:03.795214 31780 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:03.798406 31780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8858c00785e418ead2c5908393c0b9c: Generated new TSK 0
I20260812 14:27:03.798573 31780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:27:03.799585 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:03.792538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51393 (local address 127.5.148.62:40383)
0812 14:27:03.792605 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:03.792607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:03.792791 (+ 184us) server_negotiation.cc:407] Connection header received
0812 14:27:03.792869 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:03.792870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:03.792880 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:03.792946 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:03.793584 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:03.794327 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:03.795111 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:03.795397 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:03.795478 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:03.796824 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:03.796825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:03.796826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:03.796833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:03.798371 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:03.798793 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:03.798794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:03.798795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:03.798856 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:03.799065 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:03.799066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:03.799067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:03.799173 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:03.799275 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:03.799391 (+ 116us) server_negotiation.cc:299] Negotiation successful
0812 14:27:03.799491 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:27:03.801575 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 31685
2026-08-12T14:27:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:03 dist-test-slave-8w9v krb5kdc[31799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:03 dist-test-slave-8w9v krb5kdc[31799](info): set up 2 sockets
Aug 12 14:27:03 dist-test-slave-8w9v krb5kdc[31799](info): commencing operation
krb5kdc: starting...
W20260812 14:27:04.929670 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:27:05 dist-test-slave-8w9v krb5kdc[31799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:27:05.057153 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:44809
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:44809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:05.142067 31813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:05.142333 31813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:05.142374 31813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:05.143617 31813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:05.143692 31813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:05.143716 31813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:05.143739 31813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:05.143770 31813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:05.145331 31813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:44809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:44809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:05.145620 31813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:05.145862 31813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:05.151032 31822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:05.151115 31813 server_base.cc:1034] running on GCE node
W20260812 14:27:05.151072 31820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:05.151041 31819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:05.151502 31813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:05.151844 31813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:05.153038 31813 hybrid_clock.cc:648] HybridClock initialized: now 1786544825153015 us; error 35 us; skew 500 ppm
Aug 12 14:27:05 dist-test-slave-8w9v krb5kdc[31799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544825, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:05.155656 31813 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:27:05.156188 31813 webserver.cc:460] Webserver started at http://127.5.148.62:44505/ using document root <none> and password file <none>
I20260812 14:27:05.156418 31813 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:05.156488 31813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:05.156649 31813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:05.157783 31813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "eefd4cf4a01845b48a0a10ca9631c26d"
format_stamp: "Formatted at 2026-08-12 14:27:05 on dist-test-slave-8w9v"
server_key: "3526a4bbfabe53fe06546e2ac6ad3a46"
server_key_iv: "6c6a89098cee3af91b08162b07926acc"
server_key_version: "encryptionkey@0"
I20260812 14:27:05.158174 31813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "eefd4cf4a01845b48a0a10ca9631c26d"
format_stamp: "Formatted at 2026-08-12 14:27:05 on dist-test-slave-8w9v"
server_key: "3526a4bbfabe53fe06546e2ac6ad3a46"
server_key_iv: "6c6a89098cee3af91b08162b07926acc"
server_key_version: "encryptionkey@0"
I20260812 14:27:05.159574 31813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:05.160379 31831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:05.160555 31813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:05.160665 31813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "eefd4cf4a01845b48a0a10ca9631c26d"
format_stamp: "Formatted at 2026-08-12 14:27:05 on dist-test-slave-8w9v"
server_key: "3526a4bbfabe53fe06546e2ac6ad3a46"
server_key_iv: "6c6a89098cee3af91b08162b07926acc"
server_key_version: "encryptionkey@0"
I20260812 14:27:05.160745 31813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:05.190435 31813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:05.192687 31813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:05.192898 31813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:05.197363 31813 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:44809
I20260812 14:27:05.197414 31893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:44809 every 8 connection(s)
I20260812 14:27:05.197829 31813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:05.198946 31894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:05.201459 31894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d: Bootstrap starting.
I20260812 14:27:05.202095 31894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:05.202417 31894 log.cc:824] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:05.203193 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 31813
I20260812 14:27:05.203320 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:27:05.203368 31894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d: No bootstrap required, opened a new log
I20260812 14:27:05.203538 5712 external_mini_cluster.cc:1428] Setting key 1f0c8e91d09479d42c7e4400ec87106c
I20260812 14:27:05.206671 31894 raft_consensus.cc:348] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefd4cf4a01845b48a0a10ca9631c26d" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 44809 } }
I20260812 14:27:05.206848 31894 raft_consensus.cc:374] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:05.206879 31894 raft_consensus.cc:729] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eefd4cf4a01845b48a0a10ca9631c26d, State: Initialized, Role: FOLLOWER
I20260812 14:27:05.207010 31894 consensus_queue.cc:260] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefd4cf4a01845b48a0a10ca9631c26d" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 44809 } }
I20260812 14:27:05.207108 31894 raft_consensus.cc:388] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:05.207168 31894 raft_consensus.cc:482] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:05.207231 31894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:05.207885 31894 raft_consensus.cc:504] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefd4cf4a01845b48a0a10ca9631c26d" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 44809 } }
I20260812 14:27:05.208001 31894 leader_election.cc:302] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eefd4cf4a01845b48a0a10ca9631c26d; no voters:
I20260812 14:27:05.208182 31894 leader_election.cc:288] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:05.208235 31900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:05.208463 31894 sys_catalog.cc:564] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:05.208447 31900 raft_consensus.cc:686] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [term 1 LEADER]: Becoming Leader. State: Replica: eefd4cf4a01845b48a0a10ca9631c26d, State: Running, Role: LEADER
I20260812 14:27:05.208643 31900 consensus_queue.cc:237] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefd4cf4a01845b48a0a10ca9631c26d" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 44809 } }
I20260812 14:27:05.209062 31901 sys_catalog.cc:455] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eefd4cf4a01845b48a0a10ca9631c26d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefd4cf4a01845b48a0a10ca9631c26d" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 44809 } } }
I20260812 14:27:05.209241 31902 sys_catalog.cc:455] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [sys.catalog]: SysCatalogTable state changed. Reason: New leader eefd4cf4a01845b48a0a10ca9631c26d. Latest consensus state: current_term: 1 leader_uuid: "eefd4cf4a01845b48a0a10ca9631c26d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefd4cf4a01845b48a0a10ca9631c26d" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 44809 } } }
I20260812 14:27:05.209524 31902 sys_catalog.cc:458] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:05.209159 31901 sys_catalog.cc:458] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:05.210276 31911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:05.210503 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:05.204115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38377 (local address 127.5.148.62:44809)
0812 14:27:05.204369 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:05.204380 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:05.204414 (+ 34us) server_negotiation.cc:407] Connection header received
0812 14:27:05.205089 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:05.205099 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:05.205167 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:05.205328 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:05.205510 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:05.206942 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:05.207880 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:05.208561 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:05.208671 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:05.209533 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:05.209537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:05.209538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:05.209772 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:05.209816 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:05.209831 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:05.209898 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:05.210117 (+ 219us) server_negotiation.cc:299] Negotiation successful
0812 14:27:05.210183 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
I20260812 14:27:05.210927 31911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:05.211506 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:27:05.212637 31911 catalog_manager.cc:1347] Generated new cluster ID: e92990a488324a8882defcd5149ca825
I20260812 14:27:05.212689 31911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:05.215916 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:05.213077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52707 (local address 127.5.148.62:44809)
0812 14:27:05.213114 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:05.213117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:05.213292 (+ 175us) server_negotiation.cc:407] Connection header received
0812 14:27:05.213339 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:05.213340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:05.213350 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:05.213407 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:05.213549 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:05.214017 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:05.214696 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:05.214910 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:05.214996 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:05.215630 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:05.215632 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:05.215632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:05.215637 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:05.215662 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:05.215667 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:05.215719 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:05.215805 (+ 86us) server_negotiation.cc:299] Negotiation successful
0812 14:27:05.215826 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:27:05.221267 31911 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:05.222025 31911 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:05.226321 31911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eefd4cf4a01845b48a0a10ca9631c26d: Generated new TSK 0
I20260812 14:27:05.226521 31911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:27:05.231863 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 31813
2026-08-12T14:27:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:05 dist-test-slave-8w9v krb5kdc[31928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:05 dist-test-slave-8w9v krb5kdc[31928](info): set up 2 sockets
Aug 12 14:27:05 dist-test-slave-8w9v krb5kdc[31928](info): commencing operation
krb5kdc: starting...
W20260812 14:27:06.364173 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:27:06 dist-test-slave-8w9v krb5kdc[31928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:27:06.496739 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:42387
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:36283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:06.581733 31941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:06.582002 31941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:06.582046 31941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:06.583303 31941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:06.583379 31941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:06.583405 31941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:06.583425 31941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:06.583452 31941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:06.585044 31941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:36283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:42387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:42387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:06.585338 31941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:06.585580 31941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:06.590819 31950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:06.590827 31947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:06.590827 31948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:06.591084 31941 server_base.cc:1034] running on GCE node
I20260812 14:27:06.591420 31941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:06.591764 31941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:06.592941 31941 hybrid_clock.cc:648] HybridClock initialized: now 1786544826592924 us; error 43 us; skew 500 ppm
Aug 12 14:27:06 dist-test-slave-8w9v krb5kdc[31928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544826, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:06.595635 31941 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:27:06.596254 31941 webserver.cc:460] Webserver started at http://127.5.148.62:38789/ using document root <none> and password file <none>
I20260812 14:27:06.596510 31941 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:06.596582 31941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:06.596745 31941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:06.597888 31941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "3a2f54ae40a544cf8b9b8fdd59987bbb"
format_stamp: "Formatted at 2026-08-12 14:27:06 on dist-test-slave-8w9v"
server_key: "6862da5e59db6059c3adffc953c8cea5"
server_key_iv: "7d258b14c73cf78e49300fe1c93775d5"
server_key_version: "encryptionkey@0"
I20260812 14:27:06.598279 31941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "3a2f54ae40a544cf8b9b8fdd59987bbb"
format_stamp: "Formatted at 2026-08-12 14:27:06 on dist-test-slave-8w9v"
server_key: "6862da5e59db6059c3adffc953c8cea5"
server_key_iv: "7d258b14c73cf78e49300fe1c93775d5"
server_key_version: "encryptionkey@0"
I20260812 14:27:06.599790 31941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:27:06.600629 31958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:06.600798 31941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:06.600911 31941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "3a2f54ae40a544cf8b9b8fdd59987bbb"
format_stamp: "Formatted at 2026-08-12 14:27:06 on dist-test-slave-8w9v"
server_key: "6862da5e59db6059c3adffc953c8cea5"
server_key_iv: "7d258b14c73cf78e49300fe1c93775d5"
server_key_version: "encryptionkey@0"
I20260812 14:27:06.601001 31941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:06.604578 31941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:06.606981 31941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:06.607143 31941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:06.611481 31941 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:42387
I20260812 14:27:06.611526 32021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:42387 every 8 connection(s)
I20260812 14:27:06.611886 31941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:06.612884 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 31941
I20260812 14:27:06.613024 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:27:06.613014 32023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:06.613229 5712 external_mini_cluster.cc:1428] Setting key 4248f07473f14a73e987d5e379e2e48f
I20260812 14:27:06.615908 32026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:06.613829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44009 (local address 127.5.148.62:42387)
0812 14:27:06.614053 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:06.614063 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:06.614098 (+ 35us) server_negotiation.cc:407] Connection header received
0812 14:27:06.614638 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:06.614648 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:06.614684 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:06.614843 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:06.615473 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:06.615476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:06.615477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:06.615481 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:06.615527 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:06.615541 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:06.615616 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:06.615667 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 14:27:06.615709 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":104,"threads_started":1}
I20260812 14:27:06.616691 32023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb: Bootstrap starting.
I20260812 14:27:06.617013 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:27:06.617460 32023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:06.617789 32023 log.cc:824] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:06.618597 32023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb: No bootstrap required, opened a new log
I20260812 14:27:06.619827 32026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:06.618728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51541 (local address 127.5.148.62:42387)
0812 14:27:06.618766 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:06.618767 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:06.618868 (+ 101us) server_negotiation.cc:407] Connection header received
0812 14:27:06.618932 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:06.618933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:06.618938 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:06.619013 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:06.619628 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:06.619629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:06.619629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:27:06.619633 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:06.619646 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:27:06.619649 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:06.619695 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:06.619749 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 14:27:06.619763 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:27:06.620999 32023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2f54ae40a544cf8b9b8fdd59987bbb" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 42387 } }
I20260812 14:27:06.621138 32023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:27:06.621160 32023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2f54ae40a544cf8b9b8fdd59987bbb, State: Initialized, Role: FOLLOWER
I20260812 14:27:06.621276 32023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2f54ae40a544cf8b9b8fdd59987bbb" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 42387 } }
I20260812 14:27:06.621366 32023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:27:06.621390 32023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:27:06.621421 32023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:06.621927 32023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2f54ae40a544cf8b9b8fdd59987bbb" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 42387 } }
I20260812 14:27:06.622018 32023 leader_election.cc:302] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a2f54ae40a544cf8b9b8fdd59987bbb; no voters:
I20260812 14:27:06.622145 32023 leader_election.cc:288] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:27:06.622200 32033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:06.622407 32023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:06.622437 32033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [term 1 LEADER]: Becoming Leader. State: Replica: 3a2f54ae40a544cf8b9b8fdd59987bbb, State: Running, Role: LEADER
I20260812 14:27:06.622572 32033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2f54ae40a544cf8b9b8fdd59987bbb" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 42387 } }
I20260812 14:27:06.623005 32034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a2f54ae40a544cf8b9b8fdd59987bbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2f54ae40a544cf8b9b8fdd59987bbb" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 42387 } } }
I20260812 14:27:06.623035 32035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a2f54ae40a544cf8b9b8fdd59987bbb. Latest consensus state: current_term: 1 leader_uuid: "3a2f54ae40a544cf8b9b8fdd59987bbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2f54ae40a544cf8b9b8fdd59987bbb" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 42387 } } }
I20260812 14:27:06.623170 32035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:06.623457 32043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:06.623742 32034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:06.624249 32043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:06.625869 32043 catalog_manager.cc:1347] Generated new cluster ID: f2f52bd4c6f84833912db5b7fdd53f8b
I20260812 14:27:06.625931 32043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:06.631516 32043 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:06.631707 32043 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:06.639907 32043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a2f54ae40a544cf8b9b8fdd59987bbb: Generated new TSK 0
I20260812 14:27:06.640194 32043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:27:06.643986 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 31941
2026-08-12T14:27:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1411 ms)
[----------] 18 tests from AuthTokenIssuingTest (25946 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:06 dist-test-slave-8w9v krb5kdc[32057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:06 dist-test-slave-8w9v krb5kdc[32057](info): set up 2 sockets
Aug 12 14:27:06 dist-test-slave-8w9v krb5kdc[32057](info): commencing operation
krb5kdc: starting...
W20260812 14:27:07.819015 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:27:07 dist-test-slave-8w9v krb5kdc[32057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:27:07.947319 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:33221
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:33221,127.5.148.61:35361,127.5.148.60:38147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:08.040972 32072 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:27:08.041270 32072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:08.041340 32072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:08.041414 32072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:08.042595 32072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:08.042662 32072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:08.042689 32072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:08.042713 32072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:08.042739 32072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:08.044665 32072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:33221,127.5.148.61:35361,127.5.148.60:38147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:33221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:08.044956 32072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:08.045233 32072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:08.051208 32079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:08.051215 32078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:08.051438 32072 server_base.cc:1034] running on GCE node
W20260812 14:27:08.051208 32082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:08.051872 32072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:08.052314 32072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:08.053565 32072 hybrid_clock.cc:648] HybridClock initialized: now 1786544828053548 us; error 54 us; skew 500 ppm
Aug 12 14:27:08 dist-test-slave-8w9v krb5kdc[32057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544828, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:08.057443 32072 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:27:08.058086 32072 webserver.cc:460] Webserver started at http://127.5.148.62:38911/ using document root <none> and password file <none>
I20260812 14:27:08.058426 32072 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:08.058509 32072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:08.058683 32072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:08.060356 32072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "883d9c31d9404135acfccc2587dd4e27"
format_stamp: "Formatted at 2026-08-12 14:27:08 on dist-test-slave-8w9v"
server_key: "77a853a435fb11f2db94ba978c7028d4"
server_key_iv: "47352d341d50c9319c0efce2f15f553c"
server_key_version: "encryptionkey@0"
I20260812 14:27:08.060850 32072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "883d9c31d9404135acfccc2587dd4e27"
format_stamp: "Formatted at 2026-08-12 14:27:08 on dist-test-slave-8w9v"
server_key: "77a853a435fb11f2db94ba978c7028d4"
server_key_iv: "47352d341d50c9319c0efce2f15f553c"
server_key_version: "encryptionkey@0"
I20260812 14:27:08.062923 32072 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 14:27:08.063973 32090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:08.064349 32072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:27:08.064513 32072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "883d9c31d9404135acfccc2587dd4e27"
format_stamp: "Formatted at 2026-08-12 14:27:08 on dist-test-slave-8w9v"
server_key: "77a853a435fb11f2db94ba978c7028d4"
server_key_iv: "47352d341d50c9319c0efce2f15f553c"
server_key_version: "encryptionkey@0"
I20260812 14:27:08.064626 32072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:08.097612 32072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:08.099903 32072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:08.100036 32072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:08.105218 32072 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:33221
I20260812 14:27:08.105279 32154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:33221 every 8 connection(s)
I20260812 14:27:08.105685 32072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:08.106791 32155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:08.108485 32155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:08.114074 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 32072
I20260812 14:27:08.114178 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:27:08.114413 5712 external_mini_cluster.cc:1428] Setting key 5d82798e1fd13bd8f1be90bda65a02fe
Aug 12 14:27:08 dist-test-slave-8w9v krb5kdc[32057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544828, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:27:08.119124 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.109274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33221 (local address 127.0.0.1:60110)
0812 14:27:08.109461 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.109498 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:08.109961 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:08.110325 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:08.110328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:08.110603 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:08.110776 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.110780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.111820 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.111821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.112646 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.112649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.113170 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.113171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.113233 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.113747 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:08.113755 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:08.115572 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:08.117509 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.117516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.117519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.117794 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.118088 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.118089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.118089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.118179 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.118659 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:08.118665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:08.118724 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 14:27:08.118802 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 14:27:08.118948 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
I20260812 14:27:08.119236 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.109275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.5.148.62:33221)
0812 14:27:08.110098 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.110101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.110112 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:27:08.110174 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.110175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.110190 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.110252 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.110890 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.111714 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.112728 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.113042 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.113126 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.115669 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:08.115677 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:08.115677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:08.115684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:08.117410 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.117893 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.117896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.117897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.118008 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.118251 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.118252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.118252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.118582 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:08.118675 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:08.118874 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 14:27:08.119053 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":793,"thread_start_us":104,"threads_started":1}
I20260812 14:27:08.120106 32155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:08.120601 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.120426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35361 (local address 127.0.0.1:60572)
0812 14:27:08.120500 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.120575 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 14:27:08.120759 32094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35361: connect: Connection refused (error 111)
I20260812 14:27:08.121201 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.121105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35361 (local address 127.0.0.1:60588)
0812 14:27:08.121153 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.121178 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 14:27:08.121310 32155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.61:35361: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35361: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.5.148.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.61.keytab.
Entry for principal kudu/127.5.148.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.61.keytab.
I20260812 14:27:08.142590 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.61:35361
--webserver_interface=127.5.148.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.61.keytab
--principal=kudu/127.5.148.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:33221,127.5.148.61:35361,127.5.148.60:38147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:27:08.170596 32155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } attempt: 1
I20260812 14:27:08.171180 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.171008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35361 (local address 127.0.0.1:60596)
0812 14:27:08.171105 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.171144 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 14:27:08.171679 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.171567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35361 (local address 127.0.0.1:60610)
0812 14:27:08.171624 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.171657 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 14:27:08.171816 32155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.61:35361: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35361: connect: Connection refused (error 111)
W20260812 14:27:08.227839 32162 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:27:08.228055 32162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:08.228073 32162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:08.228092 32162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:08.229228 32162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:08.229276 32162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:08.229290 32162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:08.229303 32162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:08.229315 32162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:08.230728 32162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:33221,127.5.148.61:35361,127.5.148.60:38147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.61.keytab
--principal=kudu/127.5.148.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.61:35361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.148.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:08.230939 32162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:08.231125 32162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:08.236326 32172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:08.236284 32170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:08.236500 32162 server_base.cc:1034] running on GCE node
W20260812 14:27:08.236284 32169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:08.236768 32162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:08.237068 32162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:08.238243 32162 hybrid_clock.cc:648] HybridClock initialized: now 1786544828238231 us; error 50 us; skew 500 ppm
I20260812 14:27:08.239996 32155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } attempt: 2
I20260812 14:27:08.240427 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.240283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35361 (local address 127.0.0.1:60612)
0812 14:27:08.240349 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.240387 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
Aug 12 14:27:08 dist-test-slave-8w9v krb5kdc[32057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544828, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:08.240877 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.240759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35361 (local address 127.0.0.1:60628)
0812 14:27:08.240811 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.240844 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 14:27:08.240988 32155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.61:35361: Network error: Client connection negotiation failed: client connection to 127.5.148.61:35361: connect: Connection refused (error 111)
I20260812 14:27:08.241122 32162 init.cc:377] Logged in from keytab as kudu/127.5.148.61@KRBTEST.COM (short username kudu)
I20260812 14:27:08.241587 32162 webserver.cc:460] Webserver started at http://127.5.148.61:44693/ using document root <none> and password file <none>
I20260812 14:27:08.241837 32162 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:08.241883 32162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:08.242043 32162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:08.243144 32162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/data/instance:
uuid: "b7d3334328b544eeaead4b4230bf0233"
format_stamp: "Formatted at 2026-08-12 14:27:08 on dist-test-slave-8w9v"
server_key: "118f6386f551c711663cf2a16814ad1f"
server_key_iv: "816ff754dfb2dc48597dbd9ee8062c67"
server_key_version: "encryptionkey@0"
I20260812 14:27:08.243523 32162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/wal/instance:
uuid: "b7d3334328b544eeaead4b4230bf0233"
format_stamp: "Formatted at 2026-08-12 14:27:08 on dist-test-slave-8w9v"
server_key: "118f6386f551c711663cf2a16814ad1f"
server_key_iv: "816ff754dfb2dc48597dbd9ee8062c67"
server_key_version: "encryptionkey@0"
I20260812 14:27:08.245011 32162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:08.245874 32179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:08.246127 32162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:27:08.246239 32162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/wal
uuid: "b7d3334328b544eeaead4b4230bf0233"
format_stamp: "Formatted at 2026-08-12 14:27:08 on dist-test-slave-8w9v"
server_key: "118f6386f551c711663cf2a16814ad1f"
server_key_iv: "816ff754dfb2dc48597dbd9ee8062c67"
server_key_version: "encryptionkey@0"
I20260812 14:27:08.246325 32162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:08.257006 32162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:08.259534 32162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:08.259709 32162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:08.263916 32162 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.61:35361
I20260812 14:27:08.263990 32235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.61:35361 every 8 connection(s)
I20260812 14:27:08.264377 32162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/data/info.pb
I20260812 14:27:08.265525 32236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:08.267143 32236 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:08.268288 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 32162
I20260812 14:27:08.268429 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-1/wal/instance
I20260812 14:27:08.268621 5712 external_mini_cluster.cc:1428] Setting key 3ba549acdf7bed3b4c16d88b423e8735
Aug 12 14:27:08 dist-test-slave-8w9v krb5kdc[32057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544828, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.61@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.148.60@KRBTEST.COM; defaulting to no policy
I20260812 14:27:08.278096 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.267914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60114 (local address 127.5.148.62:33221)
0812 14:27:08.268003 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.268005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.268282 (+ 277us) server_negotiation.cc:407] Connection header received
0812 14:27:08.269079 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.269080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.269091 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.269180 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.270097 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.270813 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.272343 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.272566 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.272646 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.275126 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:08.275128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:08.275128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:08.275136 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:08.276619 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.277073 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.277075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.277075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.277137 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.277411 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.277412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.277412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.277530 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:08.277602 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:08.277909 (+ 307us) server_negotiation.cc:299] Negotiation successful
0812 14:27:08.278000 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 14:27:08.278293 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.267866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33221 (local address 127.0.0.1:60114)
0812 14:27:08.268101 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.268177 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:08.268941 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:08.269293 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:08.269297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:08.269718 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:08.269981 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.269986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.270925 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.270928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.272241 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.272244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.272666 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.272667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.272800 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.273372 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:08.273380 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:08.275025 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:08.276716 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.276723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.276726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.276968 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.277208 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.277209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.277209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.277327 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.277606 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:08.277614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:08.277769 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 14:27:08.277875 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 14:27:08.278048 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":107,"threads_started":1}
I20260812 14:27:08.278987 32236 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.60@KRBTEST.COM" created.
Aug 12 14:27:08 dist-test-slave-8w9v krb5kdc[32057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544828, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.61@KRBTEST.COM for kudu/127.5.148.61@KRBTEST.COM
I20260812 14:27:08.286653 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.279258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35361 (local address 127.0.0.1:60632)
0812 14:27:08.279311 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.279324 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:08.279367 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:08.279708 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:08.279709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:08.279870 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:08.279922 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.279923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.280937 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.280938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.281431 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.281433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.281906 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.281907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.281973 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.282414 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:08.282420 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:08.283727 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:08.285563 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.285564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.285564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.285734 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.285964 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.285965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.285965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.286029 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.286321 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:08.286321 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:08.286366 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 14:27:08.286432 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 14:27:08.286535 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 14:27:08.286739 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.279368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.5.148.61:35361)
0812 14:27:08.279542 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.279546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.279563 (+ 17us) server_negotiation.cc:407] Connection header received
0812 14:27:08.279579 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.279581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.279592 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.279675 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.280006 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.280834 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.281522 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.281824 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.281905 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.283825 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:08.283842 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:08.283843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:08.283850 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:08.285460 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.285828 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.285831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.285831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.285896 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.286107 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.286108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.286108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.286250 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:08.286343 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:08.286461 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 14:27:08.286607 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260812 14:27:08.287305 32236 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:08.287655 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.287551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38147 (local address 127.0.0.1:56246)
0812 14:27:08.287591 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.287635 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 14:27:08.287824 32181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
I20260812 14:27:08.288316 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.288227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38147 (local address 127.0.0.1:56254)
0812 14:27:08.288273 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.288298 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260812 14:27:08.288471 32236 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.60:38147: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.60.keytab.
Entry for principal kudu/127.5.148.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.60.keytab.
I20260812 14:27:08.297405 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.60:38147
--webserver_interface=127.5.148.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.60.keytab
--principal=kudu/127.5.148.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:33221,127.5.148.61:35361,127.5.148.60:38147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:27:08.332183 32155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } attempt: 3
Aug 12 14:27:08 dist-test-slave-8w9v krb5kdc[32057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544828, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for kudu/127.5.148.61@KRBTEST.COM
I20260812 14:27:08.337715 32236 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } attempt: 1
I20260812 14:27:08.338397 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.338283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38147 (local address 127.0.0.1:56260)
0812 14:27:08.338327 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.338358 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260812 14:27:08.338785 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.338660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38147 (local address 127.0.0.1:56266)
0812 14:27:08.338719 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.338756 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 14:27:08.338912 32236 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.60:38147: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
I20260812 14:27:08.340309 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.332683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.5.148.61:35361)
0812 14:27:08.332759 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.332762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.332775 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:27:08.332842 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.332843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.332856 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.332930 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.333292 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.333797 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.334533 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.334774 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.334853 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.337106 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:08.337108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:08.337108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:08.337116 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:08.339168 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.339560 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.339561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.339562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.339613 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.339807 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.339807 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.339808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.339910 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:08.339962 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:08.340175 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 14:27:08.340255 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 14:27:08.340332 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.332615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35361 (local address 127.0.0.1:60636)
0812 14:27:08.332673 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.332693 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:08.332771 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:08.332945 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:08.332947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:08.333123 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:08.333195 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.333196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.333891 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.333893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.334432 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.334434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.334866 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.334868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.334960 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.335447 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:08.335452 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:08.337001 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:08.339248 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.339250 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.339250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.339484 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.339679 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.339679 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.339680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.339747 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.339974 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:08.339975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:08.340031 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 14:27:08.340104 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 14:27:08.340238 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 14:27:08.340780 32155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:08.341140 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.341027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38147 (local address 127.0.0.1:56272)
0812 14:27:08.341079 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.341114 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 14:27:08.341529 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.341433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38147 (local address 127.0.0.1:56280)
0812 14:27:08.341484 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.341510 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 14:27:08.341637 32155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.60:38147: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
I20260812 14:27:08.372774 32155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } attempt: 1
I20260812 14:27:08.373288 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.373145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38147 (local address 127.0.0.1:56284)
0812 14:27:08.373215 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.373258 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 14:27:08.373703 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.373617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38147 (local address 127.0.0.1:56298)
0812 14:27:08.373659 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.373685 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 14:27:08.373831 32155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.60:38147: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
W20260812 14:27:08.384474 32243 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:27:08.384688 32243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:08.384706 32243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:08.384727 32243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:08.385807 32243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:08.385841 32243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:08.385855 32243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:08.385869 32243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:08.385880 32243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:08.387339 32243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:33221,127.5.148.61:35361,127.5.148.60:38147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.60.keytab
--principal=kudu/127.5.148.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.60:38147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.148.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:08.387547 32243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:08.387715 32243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:08.392647 32251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:08.392663 32250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:08.392653 32253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:08.392719 32243 server_base.cc:1034] running on GCE node
I20260812 14:27:08.393074 32243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:08.393383 32243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:08.394529 32243 hybrid_clock.cc:648] HybridClock initialized: now 1786544828394506 us; error 39 us; skew 500 ppm
Aug 12 14:27:08 dist-test-slave-8w9v krb5kdc[32057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544828, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:08.397068 32243 init.cc:377] Logged in from keytab as kudu/127.5.148.60@KRBTEST.COM (short username kudu)
I20260812 14:27:08.397497 32243 webserver.cc:460] Webserver started at http://127.5.148.60:40559/ using document root <none> and password file <none>
I20260812 14:27:08.397740 32243 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:08.397810 32243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:08.397974 32243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:08.399030 32243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/data/instance:
uuid: "de22f0413d6741cdbd88bdb02055bcb4"
format_stamp: "Formatted at 2026-08-12 14:27:08 on dist-test-slave-8w9v"
server_key: "438901fe7c65aeb5658cf3ff0d13bf45"
server_key_iv: "cf92f58f8243c9aa203198a0d5404920"
server_key_version: "encryptionkey@0"
I20260812 14:27:08.399405 32243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/wal/instance:
uuid: "de22f0413d6741cdbd88bdb02055bcb4"
format_stamp: "Formatted at 2026-08-12 14:27:08 on dist-test-slave-8w9v"
server_key: "438901fe7c65aeb5658cf3ff0d13bf45"
server_key_iv: "cf92f58f8243c9aa203198a0d5404920"
server_key_version: "encryptionkey@0"
I20260812 14:27:08.400928 32243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:27:08.401664 32261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:08.401839 32243 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:08.401959 32243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/wal
uuid: "de22f0413d6741cdbd88bdb02055bcb4"
format_stamp: "Formatted at 2026-08-12 14:27:08 on dist-test-slave-8w9v"
server_key: "438901fe7c65aeb5658cf3ff0d13bf45"
server_key_iv: "cf92f58f8243c9aa203198a0d5404920"
server_key_version: "encryptionkey@0"
I20260812 14:27:08.402037 32243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:08.407083 32236 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } attempt: 2
I20260812 14:27:08.407501 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.407374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38147 (local address 127.0.0.1:56312)
0812 14:27:08.407436 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.407474 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 14:27:08.407863 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.407770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38147 (local address 127.0.0.1:56320)
0812 14:27:08.407814 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.407845 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 14:27:08.407943 32236 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.60:38147: Network error: Client connection negotiation failed: client connection to 127.5.148.60:38147: connect: Connection refused (error 111)
I20260812 14:27:08.431504 32243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:08.433673 32243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:08.433844 32243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:08.437899 32243 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.60:38147
I20260812 14:27:08.437942 32321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.60:38147 every 8 connection(s)
I20260812 14:27:08.438333 32243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/data/info.pb
I20260812 14:27:08.439349 32322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:08.441054 32322 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:08.443327 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 32243
I20260812 14:27:08.443434 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786544547303129-5712-0/minicluster-data/master-2/wal/instance
I20260812 14:27:08.443610 5712 external_mini_cluster.cc:1428] Setting key 69a32bd4564f849f4fa6d9d52739956f
Aug 12 14:27:08 dist-test-slave-8w9v krb5kdc[32057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544828, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.60@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:27:08.449018 32155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } attempt: 2
Aug 12 14:27:08 dist-test-slave-8w9v krb5kdc[32057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.60@KRBTEST.COM
I20260812 14:27:08.451793 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.441769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.5.148.62:33221)
0812 14:27:08.441820 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.441822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.442035 (+ 213us) server_negotiation.cc:407] Connection header received
0812 14:27:08.442502 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.442503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.442513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.442574 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.443205 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.443929 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.445445 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.445638 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.445719 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.448359 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:08.448361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:08.448362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:08.448369 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:08.450315 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.450750 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.450752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.450752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.450819 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.451103 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.451104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.451104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.451227 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:08.451307 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:08.451604 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 14:27:08.451702 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":19}
I20260812 14:27:08.451996 32325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.441726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:33221 (local address 127.0.0.1:60126)
0812 14:27:08.441900 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.441943 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:08.442401 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:08.442656 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:08.442659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:08.442913 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:08.443118 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.443122 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.444022 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.444023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.445344 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.445348 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.445742 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.445743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.445907 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.446455 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:08.446465 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:08.448266 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:08.450400 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.450408 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.450411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.450670 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.450890 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.450890 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.450891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.451021 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.451300 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:08.451308 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:08.451457 (+ 149us) client_negotiation.cc:769] Sending connection context
0812 14:27:08.451568 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 14:27:08.451725 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260812 14:27:08.452704 32322 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:08.453681 32327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.444027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.5.148.60:38147)
0812 14:27:08.444248 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.444252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.444268 (+ 16us) server_negotiation.cc:407] Connection header received
0812 14:27:08.444324 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.444325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.444341 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.444403 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.445148 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.445836 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.446857 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.447162 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.447254 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.450132 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:08.450141 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:08.450142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:08.450149 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:08.452314 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.452706 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.452709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.452709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.452797 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.453053 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.453053 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.453054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.453241 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:08.453375 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:08.453476 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 14:27:08.453569 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":114,"threads_started":1}
Aug 12 14:27:08 dist-test-slave-8w9v krb5kdc[32057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544828, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for kudu/127.5.148.60@KRBTEST.COM
Aug 12 14:27:08 dist-test-slave-8w9v krb5kdc[32057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544828, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.60@KRBTEST.COM for kudu/127.5.148.61@KRBTEST.COM
I20260812 14:27:08.458649 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.449336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38147 (local address 127.0.0.1:56336)
0812 14:27:08.449438 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.449449 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:08.449489 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:08.449716 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:08.449717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:08.449914 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:08.449951 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.449952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.451514 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.451515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.452338 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.452341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.452815 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.452817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.452884 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.453401 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:08.453407 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:08.454960 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:08.457457 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.457458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.457459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.457685 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.457929 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.457930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.457931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.458008 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.458307 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:08.458308 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:08.458353 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 14:27:08.458435 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 14:27:08.458531 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":2}
I20260812 14:27:08.458995 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.449381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.5.148.60:38147)
0812 14:27:08.449541 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.449543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.449550 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:27:08.449567 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.449568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.449596 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.449703 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.450035 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.451411 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.452434 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.452735 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.452804 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.455048 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:08.455049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:08.455050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:08.455057 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:08.457355 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.457773 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.457773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.457774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.457836 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.458091 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.458092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.458092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.458227 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:08.458311 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:08.458787 (+ 476us) server_negotiation.cc:299] Negotiation successful
0812 14:27:08.458866 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
Aug 12 14:27:08 dist-test-slave-8w9v krb5kdc[32057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:27:08.461431 32155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27: Bootstrap starting.
I20260812 14:27:08.461560 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.453069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.5.148.61:35361)
0812 14:27:08.453128 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.453130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.453158 (+ 28us) server_negotiation.cc:407] Connection header received
0812 14:27:08.453254 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.453256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.453269 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.453333 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.453705 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.454332 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.455353 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.455680 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.455721 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.458160 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:08.458162 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:08.458162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:08.458169 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:08.459843 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.460228 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.460229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.460229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.460293 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.460946 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.460947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.460947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.461105 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:08.461184 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:08.461344 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 14:27:08.461462 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:27:08.461649 32325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.452967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:35361 (local address 127.0.0.1:60644)
0812 14:27:08.453113 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.453124 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:08.453169 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:08.453337 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:08.453338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:08.453543 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:08.453582 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.453584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.454441 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.454441 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.455241 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.455244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.455799 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.455800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.455861 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.456310 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:08.456315 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:08.458088 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:08.459943 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.459944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.459945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.460112 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.460382 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.460383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.460384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.460449 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.461176 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:08.461177 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:08.461222 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 14:27:08.461315 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 14:27:08.461544 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260812 14:27:08.462064 32322 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:08.462296 32155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:08.462613 32155 log.cc:824] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:08.462939 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.454851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.5.148.62:33221)
0812 14:27:08.454889 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.454891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.454899 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:27:08.454936 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.454937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.454946 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.455006 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.455580 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.456304 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.457207 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.457404 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.457485 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.460092 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:08.460093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:08.460093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:08.460099 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:08.461716 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.462085 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.462086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.462086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.462148 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.462380 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.462382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.462382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.462524 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:08.462688 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:08.462752 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 14:27:08.462834 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:27:08.463881 32155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27: No bootstrap required, opened a new log
I20260812 14:27:08.466125 32155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } }
I20260812 14:27:08.466284 32155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 883d9c31d9404135acfccc2587dd4e27, State: Initialized, Role: FOLLOWER
I20260812 14:27:08.466395 32155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } }
I20260812 14:27:08.466799 32155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 14:27:08 dist-test-slave-8w9v krb5kdc[32057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544828, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.60@KRBTEST.COM for kudu/127.5.148.60@KRBTEST.COM
Aug 12 14:27:08 dist-test-slave-8w9v krb5kdc[32057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.61@KRBTEST.COM
W20260812 14:27:08.469164 32342 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:27:08.469210 32342 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:27:08.468494 32330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } } }
I20260812 14:27:08.469309 32330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:27:08.470566 32325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.462344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38147 (local address 127.0.0.1:56340)
0812 14:27:08.462398 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.462409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:08.462442 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:08.463657 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:08.463657 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:08.463818 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:08.463847 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.463848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.464573 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.464575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.465043 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.465044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.465433 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.465434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.465483 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.465919 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:08.465924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:08.467414 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:08.469383 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.469384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.469384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.469592 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.469915 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.469916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.469916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.469972 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.470268 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:08.470269 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:08.470306 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 14:27:08.470374 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 14:27:08.470460 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 14:27:08.471164 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.463440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.5.148.60:38147)
0812 14:27:08.463518 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.463520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.463530 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:27:08.463551 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.463552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.463564 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.463619 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.463925 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.464496 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.465133 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.465349 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.465425 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.467496 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:08.467498 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:08.467498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:08.467506 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:08.469265 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.469775 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.469776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.469777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.469835 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.470043 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.470043 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.470044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.470183 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:08.470242 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:08.470982 (+ 740us) server_negotiation.cc:299] Negotiation successful
0812 14:27:08.471064 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:27:08.472486 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.463587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.5.148.61:35361)
0812 14:27:08.463624 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.463626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.463633 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:27:08.463655 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.463655 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.463663 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.463718 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.464354 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.465028 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.465966 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.466288 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.466360 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.469237 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:08.469239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:08.469239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:08.469247 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:08.470999 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.471483 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.471484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.471484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.471540 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.471778 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.471779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.471779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.471930 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:08.472049 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:08.472298 (+ 249us) server_negotiation.cc:299] Negotiation successful
0812 14:27:08.472380 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:27:08.470244 32072 master_runner.cc:186] Error getting master registration for 127.5.148.61:35361: OK, instance_id { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" instance_seqno: 1786544828262739 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:27:08.472673 32072 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:27:08.472919 32322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4: Bootstrap starting.
I20260812 14:27:08.473752 32322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:08.474109 32322 log.cc:824] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:08.474520 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:27:08.475095 32322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4: No bootstrap required, opened a new log
I20260812 14:27:08.475641 32122 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "883d9c31d9404135acfccc2587dd4e27"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60130
I20260812 14:27:08.475759 32122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 14:27:08.475802 32122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:08.476786 32122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } }
I20260812 14:27:08.476802 32322 raft_consensus.cc:348] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } }
I20260812 14:27:08.476998 32322 raft_consensus.cc:729] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de22f0413d6741cdbd88bdb02055bcb4, State: Initialized, Role: FOLLOWER
I20260812 14:27:08.477056 32122 leader_election.cc:288] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [CANDIDATE]: Term 1 election: Requested vote from peers b7d3334328b544eeaead4b4230bf0233 (127.5.148.61:35361), de22f0413d6741cdbd88bdb02055bcb4 (127.5.148.60:38147)
I20260812 14:27:08.477104 32322 consensus_queue.cc:260] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } }
I20260812 14:27:08.477420 32322 sys_catalog.cc:564] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:08.477442 32293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "883d9c31d9404135acfccc2587dd4e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "de22f0413d6741cdbd88bdb02055bcb4"
I20260812 14:27:08.477548 32293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:08.478469 32293 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 883d9c31d9404135acfccc2587dd4e27 in term 1.
I20260812 14:27:08.479065 32093 leader_election.cc:302] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 883d9c31d9404135acfccc2587dd4e27, de22f0413d6741cdbd88bdb02055bcb4; no voters:
I20260812 14:27:08.479245 32330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:08.479404 32330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [term 1 LEADER]: Becoming Leader. State: Replica: 883d9c31d9404135acfccc2587dd4e27, State: Running, Role: LEADER
I20260812 14:27:08.479494 32330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } }
I20260812 14:27:08.480616 32346 sys_catalog.cc:455] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } } }
I20260812 14:27:08.480903 32346 sys_catalog.cc:458] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:27:08.481508 32369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 883d9c31d9404135acfccc2587dd4e27. Latest consensus state: current_term: 1 leader_uuid: "883d9c31d9404135acfccc2587dd4e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } } }
I20260812 14:27:08.481606 32369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:08.482784 32373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:08.483232 32373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:08.482759 32243 master_runner.cc:186] Error getting master registration for 127.5.148.61:35361: OK, instance_id { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" instance_seqno: 1786544828262739 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:27:08.485401 32243 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 14:27:08.485562 32367 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:27:08.485620 32367 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:27:08.488871 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.478491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.5.148.61:35361)
0812 14:27:08.478539 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.478542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.480006 (+ 1464us) server_negotiation.cc:407] Connection header received
0812 14:27:08.480168 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.480170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.480180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.480213 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.481373 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.482119 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.483105 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.484012 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.484061 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.485734 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:08.485735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:08.485735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:08.485742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:08.487499 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.487909 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.487911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.487912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.488000 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.488242 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.488243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.488244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.488417 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:08.488523 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:08.488678 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 14:27:08.488768 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:27:08.491010 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.479918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60138 (local address 127.5.148.62:33221)
0812 14:27:08.479972 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.479974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.479999 (+ 25us) server_negotiation.cc:407] Connection header received
0812 14:27:08.480022 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.480023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.480033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.480081 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.481770 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.482438 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.486366 (+ 3928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.486669 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.486735 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.488150 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:08.488152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:08.488152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:08.488160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:08.489867 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.490227 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.490228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.490229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.490284 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.490505 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.490506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.490506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.490621 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:08.490747 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:08.490832 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 14:27:08.490915 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:27:08.493996 32293 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4 [term 1 FOLLOWER]: Refusing update from remote peer 883d9c31d9404135acfccc2587dd4e27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:08.494376 32369 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:08.496287 32346 sys_catalog.cc:455] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 883d9c31d9404135acfccc2587dd4e27. Latest consensus state: current_term: 1 leader_uuid: "883d9c31d9404135acfccc2587dd4e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } } }
I20260812 14:27:08.496405 32346 sys_catalog.cc:458] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:27:08.496838 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.479373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56352 (local address 127.5.148.60:38147)
0812 14:27:08.479603 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.479605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.484262 (+ 4657us) server_negotiation.cc:407] Connection header received
0812 14:27:08.484324 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.484325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.484335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.484383 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.485641 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.486300 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.487393 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.487704 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.487766 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.493658 (+ 5892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:08.493659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:08.493660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:08.493667 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:08.495457 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.495866 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.495868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.495868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.495933 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.496182 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.496183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.496183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.496336 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:08.496439 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:08.496666 (+ 227us) server_negotiation.cc:299] Negotiation successful
0812 14:27:08.496747 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260812 14:27:08.499023 32236 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } attempt: 3
Aug 12 14:27:08 dist-test-slave-8w9v krb5kdc[32057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544828, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.61@KRBTEST.COM for kudu/127.5.148.60@KRBTEST.COM
I20260812 14:27:08.506505 32330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "883d9c31d9404135acfccc2587dd4e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } } }
I20260812 14:27:08.506625 32330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:08.506767 32346 sys_catalog.cc:455] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "883d9c31d9404135acfccc2587dd4e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } } }
I20260812 14:27:08.506860 32346 sys_catalog.cc:458] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:27:08.507083 32377 mvcc.cc:205] Tried to move back new op lower bound from 7317687617508007936 to 7317687617452670976. Current Snapshot: MvccSnapshot[applied={T|T < 7317687617508007936}]
I20260812 14:27:08.507412 32376 mvcc.cc:205] Tried to move back new op lower bound from 7317687617508007936 to 7317687617452670976. Current Snapshot: MvccSnapshot[applied={T|T < 7317687617508007936}]
I20260812 14:27:08.507862 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.499282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:38147 (local address 127.0.0.1:56360)
0812 14:27:08.499330 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:08.499341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:08.499402 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:08.499577 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:08.499578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:08.499725 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:08.499764 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.499766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.500901 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.500902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.501745 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:08.501747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.502200 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.502201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:08.502277 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.502710 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:08.502714 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:08.504309 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:08.506295 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.506296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.506296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.506547 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.506764 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:08.506765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:08.506766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:08.506830 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:08.507517 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:08.507519 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:08.507585 (+ 66us) client_negotiation.cc:769] Sending connection context
0812 14:27:08.507656 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 14:27:08.507762 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 14:27:08.508329 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:08.499331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56360 (local address 127.5.148.60:38147)
0812 14:27:08.499472 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.499474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.499480 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:27:08.499499 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.499500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.499510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.499558 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.499856 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.500599 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.501836 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:08.502128 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:08.502188 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:08.504394 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:08.504395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:08.504395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:08.504402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:08.506206 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.506633 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.506634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.506634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.506694 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:08.506948 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:08.506949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:08.506949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:08.507115 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:08.507215 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:08.507708 (+ 493us) server_negotiation.cc:299] Negotiation successful
0812 14:27:08.507827 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260812 14:27:08.509796 32236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233: Bootstrap starting.
I20260812 14:27:08.510584 32236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:08.510910 32236 log.cc:824] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:08.512833 32373 catalog_manager.cc:1347] Generated new cluster ID: 3ea5c1ef06824521820705ea3ec48b6b
I20260812 14:27:08.512890 32373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:08.529217 32236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233: No bootstrap required, opened a new log
I20260812 14:27:08.531464 32236 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } }
I20260812 14:27:08.531637 32236 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7d3334328b544eeaead4b4230bf0233, State: Initialized, Role: FOLLOWER
I20260812 14:27:08.531733 32236 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } }
I20260812 14:27:08.532152 32236 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:08.533634 32207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:08.534621 32207 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233 [term 1 FOLLOWER]: Refusing update from remote peer 883d9c31d9404135acfccc2587dd4e27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:08.535020 32380 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:08.536875 32384 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "883d9c31d9404135acfccc2587dd4e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } } }
I20260812 14:27:08.536988 32384 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:27:08.537055 32384 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "883d9c31d9404135acfccc2587dd4e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } } }
I20260812 14:27:08.537108 32384 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:27:08.538036 32398 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:27:08.538100 32398 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:27:08.538218 32399 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 883d9c31d9404135acfccc2587dd4e27. Latest consensus state: current_term: 1 leader_uuid: "883d9c31d9404135acfccc2587dd4e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } } }
I20260812 14:27:08.538292 32399 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:27:08.538429 32373 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:08.539532 32373 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:08.544847 32369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "883d9c31d9404135acfccc2587dd4e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883d9c31d9404135acfccc2587dd4e27" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 33221 } } peers { permanent_uuid: "b7d3334328b544eeaead4b4230bf0233" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 35361 } } peers { permanent_uuid: "de22f0413d6741cdbd88bdb02055bcb4" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 38147 } } }
I20260812 14:27:08.544880 32209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "883d9c31d9404135acfccc2587dd4e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b7d3334328b544eeaead4b4230bf0233"
I20260812 14:27:08.544961 32369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:08.545009 32209 raft_consensus.cc:2396] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 883d9c31d9404135acfccc2587dd4e27 for term 1 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260812 14:27:08.545419 32400 mvcc.cc:205] Tried to move back new op lower bound from 7317687617508007936 to 7317687617452670976. Current Snapshot: MvccSnapshot[applied={T|T < 7317687617508007936}]
I20260812 14:27:08.547489 32373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 883d9c31d9404135acfccc2587dd4e27: Generated new TSK 0
I20260812 14:27:08.547691 32373 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:27:08.579371 32157 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:08.577650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.5.148.62:33221)
0812 14:27:08.577695 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.577697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.578660 (+ 963us) server_negotiation.cc:407] Connection header received
0812 14:27:08.578772 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.578774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.578786 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.578816 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.579299 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60144: BlockingRecv error: recv got EOF from 127.0.0.1:60144 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260812 14:27:08.580215 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:08.578066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.5.148.61:35361)
0812 14:27:08.578107 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.578109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.579627 (+ 1518us) server_negotiation.cc:407] Connection header received
0812 14:27:08.579726 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.579727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.579738 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.579768 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.580099 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60672: BlockingRecv error: recv got EOF from 127.0.0.1:60672 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260812 14:27:08.580946 32328 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:08.578388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56364 (local address 127.5.148.60:38147)
0812 14:27:08.578429 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.578431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.580439 (+ 2008us) server_negotiation.cc:407] Connection header received
0812 14:27:08.580537 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.580538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.580549 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.580578 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.580873 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56364: BlockingRecv error: recv got EOF from 127.0.0.1:56364 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 14:27:08.582929 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:08.582184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.5.148.61:35361)
0812 14:27:08.582215 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.582217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.582427 (+ 210us) server_negotiation.cc:407] Connection header received
0812 14:27:08.582530 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.582531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.582548 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.582577 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.582873 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60686: BlockingRecv error: recv got EOF from 127.0.0.1:60686 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 14:27:08.585767 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:08.584999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.5.148.61:35361)
0812 14:27:08.585031 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.585033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.585266 (+ 233us) server_negotiation.cc:407] Connection header received
0812 14:27:08.585369 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.585369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.585379 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.585424 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.585726 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60688: BlockingRecv error: recv got EOF from 127.0.0.1:60688 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260812 14:27:08.589534 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:08.588825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.5.148.61:35361)
0812 14:27:08.588857 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.588859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.589088 (+ 229us) server_negotiation.cc:407] Connection header received
0812 14:27:08.589175 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.589176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.589185 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.589213 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.589497 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60700: BlockingRecv error: recv got EOF from 127.0.0.1:60700 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 14:27:08.595451 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:08.594618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.5.148.61:35361)
0812 14:27:08.594653 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.594655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.594919 (+ 264us) server_negotiation.cc:407] Connection header received
0812 14:27:08.595022 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.595023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.595034 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.595064 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.595403 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60708: BlockingRecv error: recv got EOF from 127.0.0.1:60708 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260812 14:27:08.605773 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:08.604793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.5.148.61:35361)
0812 14:27:08.604834 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.604836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.605171 (+ 335us) server_negotiation.cc:407] Connection header received
0812 14:27:08.605285 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.605287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.605299 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.605330 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.605722 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60720: BlockingRecv error: recv got EOF from 127.0.0.1:60720 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260812 14:27:08.624228 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:08.623174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60724 (local address 127.5.148.61:35361)
0812 14:27:08.623220 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.623223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.623533 (+ 310us) server_negotiation.cc:407] Connection header received
0812 14:27:08.623646 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.623648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.623661 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.623693 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.624165 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60724: BlockingRecv error: recv got EOF from 127.0.0.1:60724 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260812 14:27:08.658746 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:08.657611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.5.148.61:35361)
0812 14:27:08.657664 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.657666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.658012 (+ 346us) server_negotiation.cc:407] Connection header received
0812 14:27:08.658143 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.658144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.658159 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.658196 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.658666 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60740: BlockingRecv error: recv got EOF from 127.0.0.1:60740 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260812 14:27:08.725014 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:08.724229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.5.148.61:35361)
0812 14:27:08.724302 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.724305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.724420 (+ 115us) server_negotiation.cc:407] Connection header received
0812 14:27:08.724502 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.724503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.724514 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.724569 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.724943 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60748: BlockingRecv error: recv got EOF from 127.0.0.1:60748 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260812 14:27:08.855007 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:08.854242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.5.148.61:35361)
0812 14:27:08.854312 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:08.854314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:08.854515 (+ 201us) server_negotiation.cc:407] Connection header received
0812 14:27:08.854567 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:08.854568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:08.854580 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:08.854633 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:08.854957 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60760: BlockingRecv error: recv got EOF from 127.0.0.1:60760 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260812 14:27:09.113036 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:09.112278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60770 (local address 127.5.148.61:35361)
0812 14:27:09.112368 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:09.112370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:09.112426 (+ 56us) server_negotiation.cc:407] Connection header received
0812 14:27:09.112517 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:09.112518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:09.112529 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:09.112584 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:09.112972 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60770: BlockingRecv error: recv got EOF from 127.0.0.1:60770 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 14:27:09.486198 32367 catalog_manager.cc:1259] Loaded cluster ID: 3ea5c1ef06824521820705ea3ec48b6b
I20260812 14:27:09.486284 32367 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4: loading cluster ID for follower catalog manager: success
I20260812 14:27:09.487075 32367 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4: acquiring CA information for follower catalog manager: success
I20260812 14:27:09.487355 32367 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de22f0413d6741cdbd88bdb02055bcb4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:27:09.538559 32398 catalog_manager.cc:1259] Loaded cluster ID: 3ea5c1ef06824521820705ea3ec48b6b
I20260812 14:27:09.538643 32398 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233: loading cluster ID for follower catalog manager: success
I20260812 14:27:09.539415 32398 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233: acquiring CA information for follower catalog manager: success
I20260812 14:27:09.539669 32398 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7d3334328b544eeaead4b4230bf0233: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:27:09.629051 32480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:09.626366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60776 (local address 127.5.148.61:35361)
0812 14:27:09.626514 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:09.626516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:09.626527 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:27:09.626554 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:09.626555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:09.626610 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:09.626688 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:09.626931 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:09.627426 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:09.628281 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:09.628497 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:09.628549 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:09.628728 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:09.628838 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:09.628900 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:09.628965 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 14:27:09.628982 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260812 14:27:09.633769 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:09.631077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.5.148.60:38147)
0812 14:27:09.631205 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:09.631206 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:09.631259 (+ 53us) server_negotiation.cc:407] Connection header received
0812 14:27:09.631345 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:09.631345 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:09.631382 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:09.631456 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:09.631613 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:09.632094 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:09.632978 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:09.633183 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:09.633237 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:09.633385 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:09.633488 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:09.633593 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:09.633647 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 14:27:09.633669 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260812 14:27:09.638715 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:09.635646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.5.148.60:38147)
0812 14:27:09.635678 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:09.635680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:09.635689 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:27:09.635711 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:09.635712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:09.635721 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:09.635779 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:09.635942 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:09.636591 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:09.637446 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:09.637732 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:09.637800 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:09.638345 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:09.638368 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:09.638434 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:09.638596 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 14:27:09.638621 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:27:09.639003 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:09.635478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.5.148.62:33221)
0812 14:27:09.635730 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:09.635732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:09.635869 (+ 137us) server_negotiation.cc:407] Connection header received
0812 14:27:09.635924 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:09.635925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:09.635980 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:09.636100 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:09.636244 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:09.637033 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:09.637960 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:09.638250 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:09.638302 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:09.638543 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:09.638640 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:09.638721 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:09.638901 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 14:27:09.638929 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":51,"threads_started":1}
I20260812 14:27:09.639345 32480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:09.635496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.5.148.61:35361)
0812 14:27:09.635576 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:09.635578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:09.635706 (+ 128us) server_negotiation.cc:407] Connection header received
0812 14:27:09.635783 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:09.635784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:09.635794 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:09.635855 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:09.636349 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:09.637068 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:09.638143 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:09.638392 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:09.638501 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:09.638719 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:09.638753 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:09.638805 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:09.639221 (+ 416us) server_negotiation.cc:299] Negotiation successful
0812 14:27:09.639254 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 14:27:09.640779 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 32072
I20260812 14:27:09.651461 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 32162
I20260812 14:27:09.660255 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 32243
2026-08-12T14:27:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3013 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:27:09 dist-test-slave-8w9v krb5kdc[32499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:27:09 dist-test-slave-8w9v krb5kdc[32499](info): set up 2 sockets
Aug 12 14:27:09 dist-test-slave-8w9v krb5kdc[32499](info): commencing operation
krb5kdc: starting...
W20260812 14:27:10.785389 5712 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/test-user.keytab.
Aug 12 14:27:10 dist-test-slave-8w9v krb5kdc[32499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:27:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:27:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
Entry for principal kudu/127.5.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab.
I20260812 14:27:10.916955 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:37499
--webserver_interface=127.5.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:37499,127.5.148.61:32835,127.5.148.60:40725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:27:11.001410 32514 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:27:11.001683 32514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:11.001716 32514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:11.001765 32514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:11.002938 32514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:11.003013 32514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:11.003038 32514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:11.003058 32514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:11.003091 32514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:11.004665 32514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:37499,127.5.148.61:32835,127.5.148.60:40725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.62.keytab
--principal=kudu/127.5.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.62:37499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:11.004977 32514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:11.005219 32514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:11.010267 32519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:11.010280 32520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:11.010321 32523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:11.010603 32514 server_base.cc:1034] running on GCE node
I20260812 14:27:11.010864 32514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:11.011174 32514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:11.012346 32514 hybrid_clock.cc:648] HybridClock initialized: now 1786544831012344 us; error 50 us; skew 500 ppm
Aug 12 14:27:11 dist-test-slave-8w9v krb5kdc[32499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544831, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:11.014905 32514 init.cc:377] Logged in from keytab as kudu/127.5.148.62@KRBTEST.COM (short username kudu)
I20260812 14:27:11.015340 32514 webserver.cc:460] Webserver started at http://127.5.148.62:41957/ using document root <none> and password file <none>
I20260812 14:27:11.015599 32514 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:11.015669 32514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:11.015821 32514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:11.016978 32514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/data/instance:
uuid: "ada01d2e6218428c862baca5308a8d19"
format_stamp: "Formatted at 2026-08-12 14:27:11 on dist-test-slave-8w9v"
server_key: "d6d62ce5dc852a8f79da9f66322671ed"
server_key_iv: "f6f6a1b57abc5c86b0022ba7cbaf52e1"
server_key_version: "encryptionkey@0"
I20260812 14:27:11.017343 32514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/wal/instance:
uuid: "ada01d2e6218428c862baca5308a8d19"
format_stamp: "Formatted at 2026-08-12 14:27:11 on dist-test-slave-8w9v"
server_key: "d6d62ce5dc852a8f79da9f66322671ed"
server_key_iv: "f6f6a1b57abc5c86b0022ba7cbaf52e1"
server_key_version: "encryptionkey@0"
I20260812 14:27:11.018666 32514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:27:11.019371 32530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:11.019582 32514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:11.019692 32514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/wal
uuid: "ada01d2e6218428c862baca5308a8d19"
format_stamp: "Formatted at 2026-08-12 14:27:11 on dist-test-slave-8w9v"
server_key: "d6d62ce5dc852a8f79da9f66322671ed"
server_key_iv: "f6f6a1b57abc5c86b0022ba7cbaf52e1"
server_key_version: "encryptionkey@0"
I20260812 14:27:11.019769 32514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:11.045994 32514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:11.048110 32514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:11.048362 32514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:11.052565 32514 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.62:37499
I20260812 14:27:11.052605 32593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.62:37499 every 8 connection(s)
I20260812 14:27:11.053030 32514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/data/info.pb
I20260812 14:27:11.054093 32594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:11.055761 32594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:27:11 dist-test-slave-8w9v krb5kdc[32499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544831, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:27:11.063067 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 32514
I20260812 14:27:11.063196 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-0/wal/instance
I20260812 14:27:11.063441 5712 external_mini_cluster.cc:1428] Setting key fcfc06cff6af00a553f0b54c180c5bc7
I20260812 14:27:11.065755 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.056554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37499 (local address 127.0.0.1:42018)
0812 14:27:11.056727 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.056763 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:11.057212 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:11.057483 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:11.057486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:11.057744 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:11.057920 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.057924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.058977 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.058978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.059680 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.059682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.060259 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.060261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.060318 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.060816 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:11.060825 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:11.062182 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:11.064027 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.064034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.064036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.064367 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.064602 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.064603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.064603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.064719 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.065086 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:11.065093 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:11.065169 (+ 76us) client_negotiation.cc:769] Sending connection context
0812 14:27:11.065277 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 14:27:11.065577 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":129,"thread_start_us":75,"threads_started":1}
I20260812 14:27:11.065768 32600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.056715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.5.148.62:37499)
0812 14:27:11.056848 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.056853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.056892 (+ 39us) server_negotiation.cc:407] Connection header received
0812 14:27:11.057349 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.057351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.057368 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.057450 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.058006 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.058834 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.059760 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.060080 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.060266 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.062262 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:11.062270 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:11.062270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:11.062276 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:11.063948 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.064447 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.064450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.064450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.064531 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.064784 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.064785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.064785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.065016 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:11.065084 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:11.065292 (+ 208us) server_negotiation.cc:299] Negotiation successful
0812 14:27:11.065486 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
I20260812 14:27:11.066614 32594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:11.067008 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.066872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:32835 (local address 127.0.0.1:37974)
0812 14:27:11.066934 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.066986 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260812 14:27:11.067170 32531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.148.61:32835: connect: Connection refused (error 111)
I20260812 14:27:11.067615 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.067544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:32835 (local address 127.0.0.1:37980)
0812 14:27:11.067574 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.067599 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260812 14:27:11.067745 32594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.61:32835: Network error: Client connection negotiation failed: client connection to 127.5.148.61:32835: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.5.148.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.61.keytab.
Entry for principal kudu/127.5.148.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.61.keytab.
I20260812 14:27:11.091866 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.61:32835
--webserver_interface=127.5.148.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.61.keytab
--principal=kudu/127.5.148.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:37499,127.5.148.61:32835,127.5.148.60:40725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:27:11.116942 32594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } attempt: 1
I20260812 14:27:11.117484 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.117317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:32835 (local address 127.0.0.1:37994)
0812 14:27:11.117391 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.117452 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 14:27:11.117841 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.117743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:32835 (local address 127.0.0.1:38002)
0812 14:27:11.117788 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.117815 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 14:27:11.117945 32594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.61:32835: Network error: Client connection negotiation failed: client connection to 127.5.148.61:32835: connect: Connection refused (error 111)
W20260812 14:27:11.176035 32605 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:27:11.176267 32605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:11.176285 32605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:11.176306 32605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:11.177474 32605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:11.177520 32605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:11.177534 32605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:11.177548 32605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:11.177560 32605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:11.179078 32605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:37499,127.5.148.61:32835,127.5.148.60:40725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.61.keytab
--principal=kudu/127.5.148.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.61:32835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.148.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:11.179330 32605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:11.179584 32605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:11.184446 32613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:11.184652 32605 server_base.cc:1034] running on GCE node
W20260812 14:27:11.184453 32611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:11.184461 32610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:11.185001 32605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:11.185256 32605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:11.186069 32594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } attempt: 2
I20260812 14:27:11.186398 32605 hybrid_clock.cc:648] HybridClock initialized: now 1786544831186375 us; error 37 us; skew 500 ppm
I20260812 14:27:11.186494 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.186340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:32835 (local address 127.0.0.1:38006)
0812 14:27:11.186398 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.186436 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 14:27:11.186851 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.186757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:32835 (local address 127.0.0.1:38018)
0812 14:27:11.186804 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.186828 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.61:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 14:27:11.186964 32594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.61:32835: Network error: Client connection negotiation failed: client connection to 127.5.148.61:32835: connect: Connection refused (error 111)
Aug 12 14:27:11 dist-test-slave-8w9v krb5kdc[32499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544831, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:11.189088 32605 init.cc:377] Logged in from keytab as kudu/127.5.148.61@KRBTEST.COM (short username kudu)
I20260812 14:27:11.189572 32605 webserver.cc:460] Webserver started at http://127.5.148.61:37031/ using document root <none> and password file <none>
I20260812 14:27:11.189824 32605 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:11.189903 32605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:11.190050 32605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:11.191133 32605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/data/instance:
uuid: "27b70772fb9a43a7a026d27b420a940e"
format_stamp: "Formatted at 2026-08-12 14:27:11 on dist-test-slave-8w9v"
server_key: "71efef7746e874aa399202d403817bad"
server_key_iv: "7ee6933afcb5dad13c5855259122853c"
server_key_version: "encryptionkey@0"
I20260812 14:27:11.191500 32605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/wal/instance:
uuid: "27b70772fb9a43a7a026d27b420a940e"
format_stamp: "Formatted at 2026-08-12 14:27:11 on dist-test-slave-8w9v"
server_key: "71efef7746e874aa399202d403817bad"
server_key_iv: "7ee6933afcb5dad13c5855259122853c"
server_key_version: "encryptionkey@0"
I20260812 14:27:11.192974 32605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260812 14:27:11.193713 32621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:11.193899 32605 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:27:11.194007 32605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/wal
uuid: "27b70772fb9a43a7a026d27b420a940e"
format_stamp: "Formatted at 2026-08-12 14:27:11 on dist-test-slave-8w9v"
server_key: "71efef7746e874aa399202d403817bad"
server_key_iv: "7ee6933afcb5dad13c5855259122853c"
server_key_version: "encryptionkey@0"
I20260812 14:27:11.194082 32605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:11.205487 32605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:11.207494 32605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:11.207647 32605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:11.211855 32605 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.61:32835
I20260812 14:27:11.211915 32684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.61:32835 every 8 connection(s)
I20260812 14:27:11.212445 32605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/data/info.pb
I20260812 14:27:11.213475 32685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:11.214977 32685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:11.218014 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 32605
I20260812 14:27:11.218106 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-1/wal/instance
I20260812 14:27:11.218324 5712 external_mini_cluster.cc:1428] Setting key 5bc5c55d6cc25e8013b828fe29ab5187
Aug 12 14:27:11 dist-test-slave-8w9v krb5kdc[32499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544831, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.61@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:27:11.225025 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.215609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37499 (local address 127.0.0.1:42022)
0812 14:27:11.215759 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.215801 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:11.216345 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:11.216618 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:11.216621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:11.216868 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:11.217082 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.217086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.217732 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.217732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.218693 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.218696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.219138 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.219139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.219267 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.219789 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:11.219798 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:11.221601 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:11.223557 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.223564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.223566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.223833 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.224046 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.224047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.224047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.224205 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.224516 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:11.224522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:11.224642 (+ 120us) client_negotiation.cc:769] Sending connection context
0812 14:27:11.224721 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 14:27:11.224847 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260812 14:27:11.225024 32600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.215642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.5.148.62:37499)
0812 14:27:11.215725 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.215727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.215889 (+ 162us) server_negotiation.cc:407] Connection header received
0812 14:27:11.216443 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.216444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.216458 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.216532 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.217157 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.217650 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.218794 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.219020 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.219107 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.221696 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:11.221697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:11.221698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:11.221707 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:11.223475 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.223907 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.223908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.223908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.223973 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.224268 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.224270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.224270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.224443 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:11.224529 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:11.224788 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 14:27:11.224888 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 14:27:11.225781 32685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.148.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.148.60@KRBTEST.COM" created.
Aug 12 14:27:11 dist-test-slave-8w9v krb5kdc[32499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544831, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.61@KRBTEST.COM for kudu/127.5.148.61@KRBTEST.COM
I20260812 14:27:11.234294 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.226091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:32835 (local address 127.0.0.1:38028)
0812 14:27:11.226152 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.226162 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:11.226203 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:11.226480 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:11.226481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:11.226625 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:11.226669 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.226670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.227523 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.227524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.228098 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.228101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.228622 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.228623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.228687 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.229074 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:11.229079 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:11.231305 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:11.233165 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.233166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.233166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.233334 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.233572 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.233573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.233573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.233635 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.233965 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:11.233966 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:11.234006 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:27:11.234078 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 14:27:11.234178 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 14:27:11.234567 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.226159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.5.148.61:32835)
0812 14:27:11.226336 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.226339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.226355 (+ 16us) server_negotiation.cc:407] Connection header received
0812 14:27:11.226372 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.226374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.226384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.226475 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.226755 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.227424 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.228213 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.228531 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.228622 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.231406 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:11.231414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:11.231415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:11.231421 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:11.233062 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.233432 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.233436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.233436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.233494 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.233696 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.233697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.233698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.233873 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:11.233986 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:11.234109 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 14:27:11.234231 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
I20260812 14:27:11.235229 32685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:11.235646 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.235494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:40725 (local address 127.0.0.1:34504)
0812 14:27:11.235549 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.235607 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260812 14:27:11.235809 32622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
I20260812 14:27:11.236282 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.236179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:40725 (local address 127.0.0.1:34514)
0812 14:27:11.236217 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.236250 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260812 14:27:11.236411 32685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.60:40725: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.148.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.60.keytab.
Entry for principal kudu/127.5.148.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.60.keytab.
I20260812 14:27:11.247020 5712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAW1J0s/build/release/bin/kudu
/tmp/dist-test-taskAW1J0s/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.60:40725
--webserver_interface=127.5.148.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.60.keytab
--principal=kudu/127.5.148.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.148.20:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.148.62:37499,127.5.148.61:32835,127.5.148.60:40725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:27:11.278126 32594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } attempt: 3
Aug 12 14:27:11 dist-test-slave-8w9v krb5kdc[32499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544831, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for kudu/127.5.148.61@KRBTEST.COM
I20260812 14:27:11.285676 32685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } attempt: 1
I20260812 14:27:11.285805 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.278564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:32835 (local address 127.0.0.1:38038)
0812 14:27:11.278619 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.278636 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:11.278712 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:11.278870 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:11.278870 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:11.279058 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:11.279130 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.279133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.279855 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.279857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.280455 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.280457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.280888 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.280889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.280984 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.281519 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:11.281524 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:11.282996 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:11.284690 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.284692 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.284692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.284874 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.285107 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.285108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.285108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.285175 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.285453 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:11.285455 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:11.285506 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 14:27:11.285594 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 14:27:11.285705 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:27:11.285805 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.278648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38038 (local address 127.5.148.61:32835)
0812 14:27:11.278728 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.278731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.278742 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:27:11.278775 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.278776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.278788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.278863 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.279231 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.279757 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.280556 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.280812 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.280890 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.283136 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:11.283138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:11.283138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:11.283149 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:11.284601 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.284971 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.284973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.284974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.285037 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.285242 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.285243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.285243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.285366 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:11.285460 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:11.285619 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 14:27:11.285708 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 14:27:11.286196 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.286073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:40725 (local address 127.0.0.1:34530)
0812 14:27:11.286143 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.286167 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260812 14:27:11.286458 32594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:11.286628 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.286491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:40725 (local address 127.0.0.1:34536)
0812 14:27:11.286553 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.286593 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 14:27:11.286787 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.286694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:40725 (local address 127.0.0.1:34544)
0812 14:27:11.286734 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.286760 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 14:27:11.286785 32685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.60:40725: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
I20260812 14:27:11.287106 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.286999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:40725 (local address 127.0.0.1:34550)
0812 14:27:11.287047 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.287080 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 14:27:11.287221 32594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.60:40725: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
I20260812 14:27:11.318383 32594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } attempt: 1
I20260812 14:27:11.318998 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.318843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:40725 (local address 127.0.0.1:34556)
0812 14:27:11.318913 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.318958 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 14:27:11.319393 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.319289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:40725 (local address 127.0.0.1:34566)
0812 14:27:11.319340 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.319368 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 14:27:11.319545 32594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.60:40725: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
W20260812 14:27:11.333662 32693 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:27:11.333941 32693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:27:11.334012 32693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:27:11.334054 32693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:27:11.335179 32693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:27:11.335245 32693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:27:11.335271 32693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:27:11.335302 32693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:27:11.335330 32693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:27:11.336898 32693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.148.20:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.148.62:37499,127.5.148.61:32835,127.5.148.60:40725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/krb5kdc/kudu_127.5.148.60.keytab
--principal=kudu/127.5.148.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.148.60:40725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.148.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 11:50:47 UTC on ccycloud.quasar-kkharb.root.comops.site
build id 4581990
I20260812 14:27:11.337173 32693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:27:11.337406 32693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:27:11.342378 32698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:11.342352 32704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:27:11.342408 32699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:27:11.342604 32693 server_base.cc:1034] running on GCE node
I20260812 14:27:11.342865 32693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:27:11.343219 32693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:27:11.344379 32693 hybrid_clock.cc:648] HybridClock initialized: now 1786544831344381 us; error 62 us; skew 500 ppm
Aug 12 14:27:11 dist-test-slave-8w9v krb5kdc[32499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544831, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:27:11.347014 32693 init.cc:377] Logged in from keytab as kudu/127.5.148.60@KRBTEST.COM (short username kudu)
I20260812 14:27:11.347483 32693 webserver.cc:460] Webserver started at http://127.5.148.60:40795/ using document root <none> and password file <none>
I20260812 14:27:11.347697 32693 fs_manager.cc:359] Metadata directory not provided
I20260812 14:27:11.347738 32693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:27:11.347831 32693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:27:11.348939 32693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/data/instance:
uuid: "622f475a6e9444d084c41997e3b8bc7c"
format_stamp: "Formatted at 2026-08-12 14:27:11 on dist-test-slave-8w9v"
server_key: "20a40d14c301b33c46ce7c5642687f85"
server_key_iv: "0701a871304782db4a1854a25c3cdca4"
server_key_version: "encryptionkey@0"
I20260812 14:27:11.349239 32693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/wal/instance:
uuid: "622f475a6e9444d084c41997e3b8bc7c"
format_stamp: "Formatted at 2026-08-12 14:27:11 on dist-test-slave-8w9v"
server_key: "20a40d14c301b33c46ce7c5642687f85"
server_key_iv: "0701a871304782db4a1854a25c3cdca4"
server_key_version: "encryptionkey@0"
I20260812 14:27:11.350503 32693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:27:11.351199 32712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:27:11.351397 32693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:27:11.351490 32693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/data,/tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/wal
uuid: "622f475a6e9444d084c41997e3b8bc7c"
format_stamp: "Formatted at 2026-08-12 14:27:11 on dist-test-slave-8w9v"
server_key: "20a40d14c301b33c46ce7c5642687f85"
server_key_iv: "0701a871304782db4a1854a25c3cdca4"
server_key_version: "encryptionkey@0"
I20260812 14:27:11.351548 32693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:27:11.354923 32685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } attempt: 2
I20260812 14:27:11.355338 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.355205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:40725 (local address 127.0.0.1:34574)
0812 14:27:11.355265 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.355298 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 14:27:11.355764 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.355643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:40725 (local address 127.0.0.1:34582)
0812 14:27:11.355697 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.355735 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260812 14:27:11.355906 32685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.148.60:40725: Network error: Client connection negotiation failed: client connection to 127.5.148.60:40725: connect: Connection refused (error 111)
I20260812 14:27:11.374619 32693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:27:11.376724 32693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:27:11.376848 32693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:27:11.380846 32693 rpc_server.cc:307] RPC server started. Bound to: 127.5.148.60:40725
I20260812 14:27:11.381002 306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.148.60:40725 every 8 connection(s)
I20260812 14:27:11.381313 32693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/data/info.pb
I20260812 14:27:11.382236 307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:27:11.383015 5712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAW1J0s/build/release/bin/kudu as pid 32693
I20260812 14:27:11.383104 5712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAW1J0s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786544547303129-5712-0/minicluster-data/master-2/wal/instance
I20260812 14:27:11.383344 5712 external_mini_cluster.cc:1428] Setting key 0a8e273ee92b99166ce4567c684255af
I20260812 14:27:11.384250 307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:27:11 dist-test-slave-8w9v krb5kdc[32499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544831, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.60@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
Aug 12 14:27:11 dist-test-slave-8w9v krb5kdc[32499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.60@KRBTEST.COM
I20260812 14:27:11.394086 32600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.385359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.5.148.62:37499)
0812 14:27:11.385433 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.385436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.385650 (+ 214us) server_negotiation.cc:407] Connection header received
0812 14:27:11.385719 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.385720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.385731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.385807 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.386269 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.386977 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.388218 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.388439 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.388531 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.390788 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:11.390789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:11.390790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:11.390796 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:11.392558 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.393076 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.393077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.393077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.393131 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.393402 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.393403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.393403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.393522 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:11.393604 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:11.393904 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 14:27:11.393992 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 14:27:11.394438 312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.385311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.62:37499 (local address 127.0.0.1:42032)
0812 14:27:11.385540 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.385584 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:11.385652 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:11.385830 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:11.385833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:11.386081 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:11.386177 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.386181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.387063 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.387064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.388096 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.388099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.388564 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.388566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.388637 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.389249 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:11.389258 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:11.390699 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:11.392639 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.392647 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.392649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.392998 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.393199 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.393200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.393200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.393333 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.393611 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:11.393619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:11.393760 (+ 141us) client_negotiation.cc:769] Sending connection context
0812 14:27:11.393868 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 14:27:11.394029 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":123,"threads_started":1}
I20260812 14:27:11.394680 32594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } attempt: 2
I20260812 14:27:11.394748 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.383890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.5.148.60:40725)
0812 14:27:11.384060 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.384068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.384093 (+ 25us) server_negotiation.cc:407] Connection header received
0812 14:27:11.384625 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.384632 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.384681 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.384800 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.385669 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.386908 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.388010 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.388436 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.388563 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.391395 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:11.391403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:11.391403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:11.391410 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:11.393472 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.393806 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.393809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.393810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.393870 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.394077 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.394078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.394078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.394257 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:11.394397 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:11.394566 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 14:27:11.394650 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260812 14:27:11.395221 307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:27:11 dist-test-slave-8w9v krb5kdc[32499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544831, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.62@KRBTEST.COM for kudu/127.5.148.60@KRBTEST.COM
Aug 12 14:27:11 dist-test-slave-8w9v krb5kdc[32499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544831, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.60@KRBTEST.COM for kudu/127.5.148.61@KRBTEST.COM
Aug 12 14:27:11 dist-test-slave-8w9v krb5kdc[32499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.62@KRBTEST.COM
I20260812 14:27:11.403411 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.395589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38044 (local address 127.5.148.61:32835)
0812 14:27:11.395642 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.395644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.395654 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:27:11.395723 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.395724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.395734 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.395784 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.396206 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.396779 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.397719 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.398046 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.398106 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.400455 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:11.400456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:11.400457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:11.400463 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:11.402227 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.402586 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.402587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.402588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.402655 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.402867 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.402868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.402869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.402990 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:11.403070 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:11.403238 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 14:27:11.403314 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:27:11.403429 312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.395485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.61:32835 (local address 127.0.0.1:38044)
0812 14:27:11.395526 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.395536 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:11.395655 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:11.395842 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:11.395842 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:11.396023 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:11.396075 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.396076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.396870 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.396871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.397638 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.397640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.398160 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.398166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.398218 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.398574 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:11.398580 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:11.400322 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:11.402323 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.402324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.402324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.402520 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.402740 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.402741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.402741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.402808 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.403075 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:11.403076 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:11.403117 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 14:27:11.403200 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 14:27:11.403295 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":1}
I20260812 14:27:11.403494 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.394949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:40725 (local address 127.0.0.1:34610)
0812 14:27:11.395049 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.395060 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:11.395102 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:11.395448 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:11.395450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:11.395625 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:11.395673 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.395676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.396635 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.396636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.397314 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.397316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.397759 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.397760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.397818 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.398511 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:11.398515 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:11.400510 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:11.402216 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.402217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.402218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.402461 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.402682 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.402683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.402683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.402748 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.403044 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:11.403046 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:11.403094 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 14:27:11.403144 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:27:11.403396 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 14:27:11.403563 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.395038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.5.148.60:40725)
0812 14:27:11.395309 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.395311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.395319 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:27:11.395336 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.395337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.395346 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.395747 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.395779 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.396567 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.397382 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.397700 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.397759 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.400610 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:11.400612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:11.400612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:11.400619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:11.402125 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.402562 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.402563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.402564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.402622 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.402820 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.402821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.402821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.402950 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:11.403236 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:11.403274 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 14:27:11.403353 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":91,"threads_started":1}
I20260812 14:27:11.404246 307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:27:11.404788 32600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.395797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.5.148.62:37499)
0812 14:27:11.395837 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.395839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.395850 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:27:11.395929 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.395930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.395939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.396007 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.396563 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.397221 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.398089 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.398374 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.398442 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.401157 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:11.401158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:11.401159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:11.401166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:11.403297 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.403878 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.403879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.403880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.403940 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.404229 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.404230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.404230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.404388 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:11.404546 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:11.404596 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 14:27:11.404671 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:27:11.405848 32594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19: Bootstrap starting.
I20260812 14:27:11.406731 32594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:11.407060 32594 log.cc:824] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:11.407919 32594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19: No bootstrap required, opened a new log
Aug 12 14:27:11 dist-test-slave-8w9v krb5kdc[32499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544831, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.60@KRBTEST.COM for kudu/127.5.148.60@KRBTEST.COM
I20260812 14:27:11.409727 32594 raft_consensus.cc:348] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } }
I20260812 14:27:11.409873 32594 raft_consensus.cc:729] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada01d2e6218428c862baca5308a8d19, State: Initialized, Role: FOLLOWER
I20260812 14:27:11.409966 32594 consensus_queue.cc:260] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } }
I20260812 14:27:11.410264 316 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } } }
I20260812 14:27:11.410347 32594 sys_catalog.cc:564] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:11.410440 316 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [sys.catalog]: This master's current role is: FOLLOWER
Aug 12 14:27:11 dist-test-slave-8w9v krb5kdc[32499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.148.61@KRBTEST.COM
I20260812 14:27:11.411679 312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.404484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:40725 (local address 127.0.0.1:34614)
0812 14:27:11.404581 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.404589 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:11.404622 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:11.404782 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:11.404783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:11.404950 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:11.404980 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.404981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.405672 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.405675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.406322 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.406324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.406717 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.406718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.406763 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.407161 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:11.407166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:11.408533 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:11.410600 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.410601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.410601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.410804 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.410989 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.410990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.410990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.411045 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.411373 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:11.411374 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:11.411412 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 14:27:11.411474 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 14:27:11.411570 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260812 14:27:11.412256 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.404551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.5.148.60:40725)
0812 14:27:11.404680 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.404681 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.404687 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:27:11.404704 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.404704 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.404712 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.404752 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.405049 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.405542 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.406398 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.406601 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.406676 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.408617 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:11.408618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:11.408618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:11.408625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:11.410517 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.410874 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.410874 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.410875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.410930 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.411109 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.411110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.411110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.411286 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:11.411350 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:11.412053 (+ 703us) server_negotiation.cc:299] Negotiation successful
0812 14:27:11.412127 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260812 14:27:11.413949 307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c: Bootstrap starting.
I20260812 14:27:11.414533 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.405864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.5.148.61:32835)
0812 14:27:11.405943 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.405945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.405957 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:27:11.406022 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.406023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.406031 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.406089 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.406664 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.407315 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.408367 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.408661 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.408739 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.411515 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:11.411516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:11.411517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:11.411523 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:11.413381 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.413724 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.413725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.413725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.413786 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.413990 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.413991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.413991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.414132 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:11.414291 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:11.414335 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 14:27:11.414422 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 14:27:11.414994 307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:11.415359 307 log.cc:824] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:11.415771 5712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:27:11.416306 307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c: No bootstrap required, opened a new log
I20260812 14:27:11.413391 32514 master_runner.cc:186] Error getting master registration for 127.5.148.61:32835: OK, instance_id { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" instance_seqno: 1786544831210591 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:27:11.416616 32514 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:27:11.416709 32564 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ada01d2e6218428c862baca5308a8d19"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42042
I20260812 14:27:11.416857 32564 raft_consensus.cc:482] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 14:27:11.416913 32564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:11.417635 32564 raft_consensus.cc:504] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } }
I20260812 14:27:11.417914 32564 leader_election.cc:288] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [CANDIDATE]: Term 1 election: Requested vote from peers 27b70772fb9a43a7a026d27b420a940e (127.5.148.61:32835), 622f475a6e9444d084c41997e3b8bc7c (127.5.148.60:40725)
I20260812 14:27:11.418401 307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } }
I20260812 14:27:11.418561 307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622f475a6e9444d084c41997e3b8bc7c, State: Initialized, Role: FOLLOWER
I20260812 14:27:11.418653 307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } }
I20260812 14:27:11.419006 307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:11.419106 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ada01d2e6218428c862baca5308a8d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "622f475a6e9444d084c41997e3b8bc7c"
I20260812 14:27:11.419245 32740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:11.420096 32740 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada01d2e6218428c862baca5308a8d19 in term 1.
I20260812 14:27:11.420408 32531 leader_election.cc:302] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622f475a6e9444d084c41997e3b8bc7c, ada01d2e6218428c862baca5308a8d19; no voters:
I20260812 14:27:11.420532 316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:27:11.420794 316 raft_consensus.cc:686] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [term 1 LEADER]: Becoming Leader. State: Replica: ada01d2e6218428c862baca5308a8d19, State: Running, Role: LEADER
I20260812 14:27:11.420892 316 consensus_queue.cc:237] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } }
W20260812 14:27:11.421315 351 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260812 14:27:11.421319 316 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ada01d2e6218428c862baca5308a8d19. Latest consensus state: current_term: 1 leader_uuid: "ada01d2e6218428c862baca5308a8d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } } }
I20260812 14:27:11.421406 316 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [sys.catalog]: This master's current role is: LEADER
W20260812 14:27:11.421402 351 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 14:27:11.421852 329 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:27:11.421906 329 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:27:11.422523 358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:27:11.422756 358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:27:11.424373 32740 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c [term 1 FOLLOWER]: Refusing update from remote peer ada01d2e6218428c862baca5308a8d19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:11.424667 316 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:11.423388 32693 master_runner.cc:186] Error getting master registration for 127.5.148.61:32835: OK, instance_id { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" instance_seqno: 1786544831210591 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:27:11.427873 32693 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:27:11.428309 336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ada01d2e6218428c862baca5308a8d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } } }
I20260812 14:27:11.428484 336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:27:11.428601 336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ada01d2e6218428c862baca5308a8d19. Latest consensus state: current_term: 1 leader_uuid: "ada01d2e6218428c862baca5308a8d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } } }
I20260812 14:27:11.428686 336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:27:11.432037 32600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.419225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.5.148.62:37499)
0812 14:27:11.419268 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.419270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.421480 (+ 2210us) server_negotiation.cc:407] Connection header received
0812 14:27:11.421682 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.421683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.421695 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.421726 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.422755 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.423501 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.424747 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.425028 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.425072 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.427369 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:11.427371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:11.427372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:11.427380 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:11.429858 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.430277 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.430278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.430279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.430339 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.431368 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.431369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.431370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.431533 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:11.431625 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:11.431863 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 14:27:11.431944 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:27:11.438627 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.421622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34626 (local address 127.5.148.60:40725)
0812 14:27:11.421662 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.421664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.428953 (+ 7289us) server_negotiation.cc:407] Connection header received
0812 14:27:11.429110 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.429111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.429135 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.429169 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.430202 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.430908 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.432402 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.432707 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.432750 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.435045 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:11.435047 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:11.435047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:11.435055 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:11.436904 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.437529 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.437531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.437531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.437605 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.437904 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.437905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.437906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.438059 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:11.438132 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:11.438447 (+ 315us) server_negotiation.cc:299] Negotiation successful
0812 14:27:11.438530 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:27:11.439970 316 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ada01d2e6218428c862baca5308a8d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } } }
I20260812 14:27:11.440078 316 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:11.440284 359 mvcc.cc:205] Tried to move back new op lower bound from 7317687629511823360 to 7317687629500796928. Current Snapshot: MvccSnapshot[applied={T|T < 7317687629511823360}]
I20260812 14:27:11.440439 360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ada01d2e6218428c862baca5308a8d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } } }
I20260812 14:27:11.440548 360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:27:11.440704 362 mvcc.cc:205] Tried to move back new op lower bound from 7317687629511823360 to 7317687629500796928. Current Snapshot: MvccSnapshot[applied={T|T < 7317687629511823360}]
I20260812 14:27:11.441258 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.420800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38056 (local address 127.5.148.61:32835)
0812 14:27:11.420838 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.420840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.425575 (+ 4735us) server_negotiation.cc:407] Connection header received
0812 14:27:11.425693 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.425695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.425706 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.425736 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.427697 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.428423 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.433045 (+ 4622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.433367 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.433423 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.435590 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:11.435591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:11.435591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:11.435599 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:11.437652 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.438685 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.438686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.438686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.438753 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.439291 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.439293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.439293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.439453 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:11.439547 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:11.441044 (+ 1497us) server_negotiation.cc:299] Negotiation successful
0812 14:27:11.441150 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:27:11.442461 358 catalog_manager.cc:1347] Generated new cluster ID: 7cd3e97ae6c54234b7a99f1b54e4fafb
I20260812 14:27:11.442521 358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:27:11.447024 32685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } attempt: 3
Aug 12 14:27:11 dist-test-slave-8w9v krb5kdc[32499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786544831, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.148.61@KRBTEST.COM for kudu/127.5.148.60@KRBTEST.COM
I20260812 14:27:11.456367 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.447313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.148.60:40725 (local address 127.0.0.1:34636)
0812 14:27:11.447639 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 14:27:11.447651 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:27:11.447697 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:27:11.448233 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:27:11.448234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:27:11.448416 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:27:11.448468 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.448470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.449343 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.449343 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.450160 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:27:11.450162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.450632 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.450633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:27:11.450704 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.451181 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:27:11.451186 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:27:11.452876 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:27:11.455163 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.455164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.455164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.455428 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.455638 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:27:11.455638 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:27:11.455639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:27:11.455703 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:27:11.456010 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:27:11.456011 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:27:11.456062 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 14:27:11.456157 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 14:27:11.456261 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305}
I20260812 14:27:11.456808 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:11.447367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.5.148.60:40725)
0812 14:27:11.447420 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.447423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.448112 (+ 689us) server_negotiation.cc:407] Connection header received
0812 14:27:11.448151 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.448151 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.448163 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.448223 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.448541 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.449248 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.450250 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:11.450555 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:11.450619 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:11.452972 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:27:11.452974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:27:11.452974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:27:11.452995 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:27:11.454847 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.455499 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.455500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.455501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.455578 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:27:11.455780 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:27:11.455781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:27:11.455781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:27:11.455921 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:27:11.456010 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:11.456636 (+ 626us) server_negotiation.cc:299] Negotiation successful
0812 14:27:11.456718 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:27:11.458571 32685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e: Bootstrap starting.
I20260812 14:27:11.459350 32685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e: Neither blocks nor log segments found. Creating new log.
I20260812 14:27:11.459667 32685 log.cc:824] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:27:11.468684 32685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e: No bootstrap required, opened a new log
I20260812 14:27:11.470834 32685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } }
I20260812 14:27:11.470999 32685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27b70772fb9a43a7a026d27b420a940e, State: Initialized, Role: FOLLOWER
I20260812 14:27:11.471079 32685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } }
I20260812 14:27:11.471446 32685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:27:11.471487 32654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ada01d2e6218428c862baca5308a8d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27b70772fb9a43a7a026d27b420a940e"
I20260812 14:27:11.471609 32654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:27:11.472498 32654 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada01d2e6218428c862baca5308a8d19 in term 1.
I20260812 14:27:11.472651 32652 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e [term 1 FOLLOWER]: Refusing update from remote peer ada01d2e6218428c862baca5308a8d19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:27:11.472942 352 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:27:11.474500 381 mvcc.cc:205] Tried to move back new op lower bound from 7317687629511823360 to 7317687629500796928. Current Snapshot: MvccSnapshot[applied={T|T < 7317687629511823360}]
I20260812 14:27:11.480247 367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ada01d2e6218428c862baca5308a8d19. Latest consensus state: current_term: 1 leader_uuid: "ada01d2e6218428c862baca5308a8d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } } }
I20260812 14:27:11.480369 367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:27:11.480378 383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ada01d2e6218428c862baca5308a8d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } } }
I20260812 14:27:11.480490 367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ada01d2e6218428c862baca5308a8d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } } }
I20260812 14:27:11.480559 367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:27:11.480496 383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:27:11.481233 385 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:27:11.481328 385 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:27:11.483821 352 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ada01d2e6218428c862baca5308a8d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada01d2e6218428c862baca5308a8d19" member_type: VOTER last_known_addr { host: "127.5.148.62" port: 37499 } } peers { permanent_uuid: "27b70772fb9a43a7a026d27b420a940e" member_type: VOTER last_known_addr { host: "127.5.148.61" port: 32835 } } peers { permanent_uuid: "622f475a6e9444d084c41997e3b8bc7c" member_type: VOTER last_known_addr { host: "127.5.148.60" port: 40725 } } }
I20260812 14:27:11.483932 352 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19 [sys.catalog]: This master's current role is: LEADER
I20260812 14:27:11.485219 358 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:27:11.485988 358 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:27:11.499495 358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ada01d2e6218428c862baca5308a8d19: Generated new TSK 0
I20260812 14:27:11.499787 358 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:27:11.530911 32600 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:11.529234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.5.148.62:37499)
0812 14:27:11.529280 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.529282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.530206 (+ 924us) server_negotiation.cc:407] Connection header received
0812 14:27:11.530316 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.530317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.530328 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.530360 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.530837 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42064: BlockingRecv error: recv got EOF from 127.0.0.1:42064 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 14:27:11.531801 32690 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:11.529647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38070 (local address 127.5.148.61:32835)
0812 14:27:11.529690 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.529692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.531272 (+ 1580us) server_negotiation.cc:407] Connection header received
0812 14:27:11.531372 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.531373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.531383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.531424 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.531741 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38070: BlockingRecv error: recv got EOF from 127.0.0.1:38070 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260812 14:27:11.532624 311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:11.529952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34644 (local address 127.5.148.60:40725)
0812 14:27:11.529994 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.529996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.532094 (+ 2098us) server_negotiation.cc:407] Connection header received
0812 14:27:11.532213 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.532214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.532224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.532254 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.532563 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34644: BlockingRecv error: recv got EOF from 127.0.0.1:34644 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260812 14:27:11.534906 32690 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:11.534215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38076 (local address 127.5.148.61:32835)
0812 14:27:11.534247 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.534249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.534425 (+ 176us) server_negotiation.cc:407] Connection header received
0812 14:27:11.534528 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.534529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.534547 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.534575 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.534861 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38076: BlockingRecv error: recv got EOF from 127.0.0.1:38076 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 14:27:11.537868 32690 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:11.537169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38080 (local address 127.5.148.61:32835)
0812 14:27:11.537201 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.537202 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.537410 (+ 208us) server_negotiation.cc:407] Connection header received
0812 14:27:11.537510 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.537510 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.537520 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.537547 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.537827 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38080: BlockingRecv error: recv got EOF from 127.0.0.1:38080 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 14:27:11.541757 32690 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:11.541032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.5.148.61:32835)
0812 14:27:11.541072 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.541074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.541269 (+ 195us) server_negotiation.cc:407] Connection header received
0812 14:27:11.541355 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.541356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.541365 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.541408 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.541717 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38088: BlockingRecv error: recv got EOF from 127.0.0.1:38088 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 14:27:11.547628 32690 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:11.546898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38096 (local address 127.5.148.61:32835)
0812 14:27:11.546930 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.546932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.547137 (+ 205us) server_negotiation.cc:407] Connection header received
0812 14:27:11.547228 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.547229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.547239 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.547268 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.547585 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38096: BlockingRecv error: recv got EOF from 127.0.0.1:38096 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260812 14:27:11.558033 32690 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:11.557088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38106 (local address 127.5.148.61:32835)
0812 14:27:11.557136 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.557138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.557417 (+ 279us) server_negotiation.cc:407] Connection header received
0812 14:27:11.557533 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.557535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.557547 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.557578 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.557977 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38106: BlockingRecv error: recv got EOF from 127.0.0.1:38106 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260812 14:27:11.576632 32690 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:11.575617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.5.148.61:32835)
0812 14:27:11.575674 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.575676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.575868 (+ 192us) server_negotiation.cc:407] Connection header received
0812 14:27:11.575981 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.575982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.576008 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.576040 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.576572 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38120: BlockingRecv error: recv got EOF from 127.0.0.1:38120 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260812 14:27:11.611104 32690 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:11.610167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38132 (local address 127.5.148.61:32835)
0812 14:27:11.610215 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.610217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.610471 (+ 254us) server_negotiation.cc:407] Connection header received
0812 14:27:11.610576 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.610577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.610590 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.610620 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.611038 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38132: BlockingRecv error: recv got EOF from 127.0.0.1:38132 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 14:27:11.677532 32690 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:11.676748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.5.148.61:32835)
0812 14:27:11.676822 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.676825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.676956 (+ 131us) server_negotiation.cc:407] Connection header received
0812 14:27:11.677020 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.677021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.677032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.677076 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.677444 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38148: BlockingRecv error: recv got EOF from 127.0.0.1:38148 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260812 14:27:11.807610 32690 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:11.806787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.5.148.61:32835)
0812 14:27:11.806881 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:11.806883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:11.806990 (+ 107us) server_negotiation.cc:407] Connection header received
0812 14:27:11.807095 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:11.807096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:11.807109 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:11.807190 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:11.807544 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38150: BlockingRecv error: recv got EOF from 127.0.0.1:38150 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 14:27:12.065630 32690 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:27:12.064840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.5.148.61:32835)
0812 14:27:12.064935 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:12.064937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:12.064962 (+ 25us) server_negotiation.cc:407] Connection header received
0812 14:27:12.065048 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:12.065049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:12.065059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:12.065126 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:27:12.065556 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38152: BlockingRecv error: recv got EOF from 127.0.0.1:38152 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 14:27:12.421947 351 catalog_manager.cc:1259] Loaded cluster ID: 7cd3e97ae6c54234b7a99f1b54e4fafb
I20260812 14:27:12.422031 351 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c: loading cluster ID for follower catalog manager: success
I20260812 14:27:12.422907 351 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c: acquiring CA information for follower catalog manager: success
I20260812 14:27:12.423236 351 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 622f475a6e9444d084c41997e3b8bc7c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:27:12.481853 385 catalog_manager.cc:1259] Loaded cluster ID: 7cd3e97ae6c54234b7a99f1b54e4fafb
I20260812 14:27:12.481950 385 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e: loading cluster ID for follower catalog manager: success
I20260812 14:27:12.482723 385 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e: acquiring CA information for follower catalog manager: success
I20260812 14:27:12.482995 385 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27b70772fb9a43a7a026d27b420a940e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:27:12.581570 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:12.578926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.5.148.61:32835)
0812 14:27:12.579031 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:12.579033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:12.579062 (+ 29us) server_negotiation.cc:407] Connection header received
0812 14:27:12.579141 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:12.579142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:12.579199 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:12.579287 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:12.579431 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:12.579934 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:12.580687 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:12.580877 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:12.580985 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:12.581190 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:12.581304 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:12.581386 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:12.581446 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 14:27:12.581469 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
I20260812 14:27:12.585711 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:12.583506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34654 (local address 127.5.148.60:40725)
0812 14:27:12.583612 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:12.583614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:12.583622 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:27:12.583679 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:12.583680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:12.583714 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:12.583782 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:12.583863 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:12.584388 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:12.585066 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:12.585252 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:12.585301 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:12.585441 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:12.585538 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:12.585594 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:12.585633 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 14:27:12.585649 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":53,"threads_started":1}
I20260812 14:27:12.590423 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:12.587347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.5.148.61:32835)
0812 14:27:12.587459 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:12.587461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:12.587574 (+ 113us) server_negotiation.cc:407] Connection header received
0812 14:27:12.587639 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:12.587640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:12.587650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:12.587703 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:12.587900 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:12.588559 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:12.589580 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:12.589864 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:12.589926 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:12.590135 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:12.590167 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:12.590220 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:12.590318 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 14:27:12.590343 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 14:27:12.590813 479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:12.587349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.5.148.62:37499)
0812 14:27:12.587458 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:12.587461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:12.587474 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:27:12.587565 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:12.587566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:12.587619 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:12.587695 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:12.588201 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:12.588909 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:12.589623 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:12.589818 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:12.589906 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:12.590120 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:12.590231 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:12.590314 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:12.590705 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 14:27:12.590732 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":65,"threads_started":1}
I20260812 14:27:12.591336 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:27:12.587696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.5.148.60:40725)
0812 14:27:12.587736 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:27:12.587737 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:27:12.587762 (+ 25us) server_negotiation.cc:407] Connection header received
0812 14:27:12.587822 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:27:12.587822 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:27:12.587833 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:27:12.587895 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:27:12.588788 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:12.589492 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:12.590583 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:27:12.590882 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:27:12.590958 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:27:12.591088 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:27:12.591118 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:27:12.591167 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:27:12.591242 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 14:27:12.591263 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:27:12.592789 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 32514
I20260812 14:27:12.603052 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 32605
I20260812 14:27:12.610478 5712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAW1J0s/build/release/bin/kudu with pid 32693
2026-08-12T14:27:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2950 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5964 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (285311 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST